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Al?gtraqt: 

In  an  effort  to  improve  the  management  feedback  in 
operational  contracting  squadrons,  this  research 
concentrated  on  the  development  of  an  alternate  method  to 
measure  operational  contracting  performance.  Specifically, 
the  research  investigated  the  use  of  Data  Envelopment 
Analysis  (DEA)  to  measure  the  productive  efficiency  of  45 
operational  contracting  squadrons.  Operational  contracting 
managers  were  surveyed  to  identify  critical  resources  to  and 
outputs  from  the  contracting  process.  Based  on  this  survey, 
four  inputs  and  five  outputs  were  included  in  the  DEA  model. 
The  DEA  was  executed  for  each  contracting  squadron  under 
study.  The  DEA  output  provided  improved  performance 
measurement  and  feedback  information.  DEA  combined  multiple 
inputs  and  outputs  into  a  single  measure  of  performance. 
Because  it  allows  flexible  weighting  of  decision  variables, 
DEA  accounted  for  differences  in  squadron  size,  mission,  and 
purchase  complexity.  By  examining  the  DEA  generated 
Hypothetical  Comparison  Unit,  specific  input  reductions  and 
output  increases  were  estedjlished  for  each  relatively 
inefficient  squadron.  Finally,  the  DEA  output  addressed 
several  desired  characteristics  of  a  performance  measurement 
system  identified  by  contracting  managers. 
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MEASURING  PRODUCTIVE  EFFICIENCY  IN 
AIR  FORCE  OPERATIONAL  CONTRACTING  SQUADRONS: 

AN  APPLICATION  OF  DATA  ENVELOPMENT  ANALYSIS 

I. 

Whether  called  contracting/  acquisition/  or  purchasing/ 
the  function  of  acquiring  goods  and  services  has  evolved 
into  a  critical  aspect  of  both  commercial  industry  and 
government  operations.  In  recent  years /  the  commercial 
sector  has  recognized  purchasing  as  an  activity  which 
contributes  directly  to  company  profits  (41:27).  With 
inevitable  decreases  in  the  military  budget/  the  Department  ’ 
of  Defense  (DOD)  has  realized  the  importance  of  improving 
efficiency  and  productivity  in  contracting  operations . 

Before  improvements  can  be  achieved/  contracting  managers 
must  measure  and  obtain  feedback  on  performance. 

The  Air  Force  conducts  purchasing  activities  in  a 
number  of  locations.  At  each  Air  Force  base/  purchasing 
activities  are  conducted  by  the  operational  contracting 
squadron.  This  squadron  is  responsible  for  acquiring  all 
goods  and  services  to  maintain  and  operate  the  base.  Base 
acquisition  may  include  simple  items  such  as  office  supplies 
and  iuel  or  more  complex  items  such  as  construction  or 
physician  support  services.  The  magnitude  of  base 
acquisition  is  quite  large.  The  Contracting  Activity  Report 
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(RCS  HAF-RDC  M&A  7106),  revealed  Air  Combat  Command's  (ACC) 
38  squadrons  executed  1,085,009  contracting  actions  valued 
at  $1,615,490,047  during  fiscal  year  1992.  ACC  is  just  one 
of  several  major  commands  in  the  Air  Force. 

General  Issue 

The  ability  to  measure  and  provide  feedback  concerning 
productivity  or  efficiency  is  importemt  to  operational 
contracting  for  several  reasons.  First,  efficiency  and 
productivity  are  major  factors  of  an  operational  contracting 
squadron's  performance.  Traditional  definitions  of 
purchasing  performance  include  the  concepts  of  effectiveness 
and  efficiency.  Effectiveness  is  difficult  to  measure  in  a 
service  function  because  it  includes  subjective  ratings  of 
goal  attainment  and  customer  satisfaction  (56:17). 

Purchasing  efficiency  and  productivity  are  closely 
related  concepts  which  can  be  measured  with  greater 
accuracy.  Purchasing  efficiency  is  the  relationship  between 
"planned  and  actual  sacrifices  made  in  order  to  realize  a 
goal  previously  agreed  upon"  (56:19).  Productivity  is  a 
measure  of  the  ratio  of  outputs  produced-  by  an  organization 
compared  to  the  inputs  used  to  create  the  output  (42:420). 

As  major  components  of  overall  purchasing  performance,  the 
ability  to  measure  efficiency  and  productivity  is  an 
important  tool  for  operational  contracting  performance 
improvement . 


Another  reason  to  measure  and  provide  feedback 
concerning  operational  contracting  perfomance  is  its 
importance  to  maintaining  national  defense.  The  operational 
contracting  squadron  provides  each  Air  Force  base  with  the 
supplies  2md  services  to  maintain  its  defense  posture. 

Given  the  current  era  of  increasing  government  deficits  and 
declining  defense  budgets,  the  need  to  measure  emd  improve 
purchasing  efficiency,  thus  reducing  operational  costs,  is 
vital  to  maintaining  national  defense  (28:46).  Just  as 
industry  is  discovering,  purchasing  plays  a  major  role  in 
achieving  Air  Force  mission  objectives. 

A  final  reason  to  measure  and  provide  feedback 
concerning  operational  contracting  performance  is  the 
current  implementation  of  the  Defense  Business  Operations 
Fund  (DBOF).  In  an  attempt  to  adapt  to  lower  budgets  and 
personnel  levels,  the  OOD  formally  estcUslished  DBOF  in 
fiscal  year  1992  (28:45).  The  long  range  goal  of  DBOF  is  to 
move  all  mission  support  activities  into  a  large  revolving 
fund  account  (53:11). 

Although  not  fully  implemented,  DBOF  attempts  to  model 
DOD  support  operations  along  the  lines  of  the  private 
sector.  Mission  support  activities,  like  operational 
contracting,  will  no  longer  receive  direct  appropriated 
fimding.  Instead,  commanders  of  direct  mission  elements 
will  receive  funds  which  may  be  used  to  purchase  support 
activities  (28:47).  These  commanders  will  then  choose  a 
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support  service  based  on  the  lowest  cost  available.  In 
effect/  DBOF  will  force  competition  for  work  among  those 
organizations  providing  services  within  the  Air  Force. 

The  price  charged  for  support  services  under  DBOF  will 
be  determined  through  unit  cost  accounting  measures.  Costs 
are  currently  being  developed  based  on  total  costs  per  unit 
of  output  (53:11).  Once  developed,  these  costs  will 
establish  the  price  mission  commanders  must  pay  for  service 
functions.  Under  DBOF,  improvements  in  contracting  squadron 
productivity  and  efficiency  will  lower  the  unit  cost  of 
service.  The  most  efficient  squadrons  may  garner  the 
majority  of  funding  or  compete  with  the  threat  of 
privatization  of  the  operational  contracting  mission. 

Current  Methods  Of  Measuring  Operational  Contracting 
Performance .  Under  current  practices  the  Air  Force  has 
limited  methods  of  measuring  how  well  the  operational 
contracting  squadron  creates  outputs  from  a  given  level  of 
resources  or  inputs.  In  other  words,  the  Air  Force  has  no 
way  to  measure  and  provide  feedback  concerning  productive 
efficiency.  The  current  method  of  overall  performance 
measurement  involves  four  separate  systems.  These  systems 
are  described  briefly  in  the  following  paragraphs,  with 
detailed  explanations  and  system  weaknesses  provided  in 
Chapter  II. 

The  first  system  of  performance  evaluation  is  the  self¬ 
inspection.  A  self-inspection  integrates  compliance  based 
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issues  into  a  checklist.  This  checklist  is  used  by  members 
of  the  operational  contracting  squadron  to  evaluate  overall 
performance.  Because  it  is  a  compliance  based  review,  the 
self-inspection  provides  limited  feedback  on  productivity 
and  efficiency.  The  main  concern  of  a  self -inspect ion  is  to 
ensure  proper  adherence  to  laws  and  regulations,  and  proper 
contract  dociunentation. 

The  second  system  is  the  use  of  the  Command  Inspector 
General  (IG)  to  conduct  a  compliance  evaluation  of  each 
squadron.  These  evaluations  are  made  every  twelve  to 
eighteen  months  cuid  last  between  seven  and  ten  days.  Like 
the  self-inspection,  the  IG  is  a  compliance  based 
inspection.  Because  of  the  extended  time  between  IG  visits, 
the  system  provides  infrequent  feedback  to  contracting 
managers . 

The  third  system  of  operational  contracting  performance 
evaluation  is  the  attainment  of  Command  and  Air  Force  goals. 
These  goals  normally  concentrate  on  competitive  purchasing 
and  socio-economic  policy.  For  example,  each  squadron  is 
judged  against  a  pre-established  small  business  usage 
standard.  These  goals  are  set  at  the  beginning  of  each 
fiscal  year.  Once  again,  feedback  provided  is 
infrequent  and  provides  little  information  on  productivity 
or  efficiency. 

The  final  system  of  performance  measurement  is  the 
contracting  manager's  ability  to  extras. t  and  compare 
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individual  measures  from  the  Base  Contracting  Automated 
System  ( BCAS ) .  These  measures  provide  individual  metrics 
for  various  facets  of  performance.  For  example,  the  BCAS 
system  provides  measures  on  total  dollars  awarded,  contract 
administrative  lead  time  (CALT),  and  the  nimber  of  purchases 
made  competitively.  Currently,  there  is  no  method  to 
aggregate  these  individual  measures  into  an  overall  index  of 
performemce.  In  addition,  the  BCAS  measures  provide  no 
indication  of  performance  relative  to  other  contracting 
squadrons . 

Data  Envelopment  Analysis:  A  Recent  Development  in 
Productivity  Measurement .  A  possible  addition  to  current 
methods  of  operational  contracting  performance  measurement 
is  the  technique  of  Data  Envelopment  Analysis  ( DEA ) .  DEA  is 
a  mathematical  progr2unming  technique  developed  by  Charnes, 
Cooper,  and  Rhodes  in  1978  (13:429).  The  technique  is 
described  in  detail  in  Chapter  III,  however  a  brief 
description  of  DEA  and  its  advantages  are  outlined  in  the 
following  paragraphs. 

An  advantage  of  DEA  is  its  5d>ility  to  establish  a 
composite  index  of  overall  performance  based  upon  relative 
productive  efficiency,  or  productivity,  of  each  operational 
contracting  squadron.  The  method  uses  traditional  simplex 
linear  programming  to  provide  an  overall  ranking  of 
performemce  for  each  member  of  a  set  of  comparable  producing 
units.  These  units  are  termed  Decision  Making  Units  (DMUs) 
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in  the  OEA  methodology.  The  use  of  DEA  is  appropriate  when 
it  is  difficult  to  place  em  economic  price  or  value  on  a 
DMU ' s  resources  or  outputs  (49:7). 

Another  adv2Uitage  of  the  DEA  methodology  is  its  ability 
to  account  for  differences  between  each  contracting 
squadron.  By  allowing  flexible  weighting  of  inputs  and 
outputs,  the  technique  ensures  differences  in  squadron 
missions,  goals,  responsibilities,  and  purchase  complexity 
are  considered  when  performance  is  evaluated.  The  ability 
to  account  for  squadron  differences  is  a  characteristic 
^d3sent  from  current  methods  of  evaluating  operational 
contracting  performance. 

The  OEA  technique  can  convert  multiple  input  and  output 
measures  into  a  single  index  of  productive  efficiency 
(22:90).  Inputs  are  the  resources  used  to  produce  a  product 
or  service.  Exeunples  of  operational  contracting  inputs  are 
the  number  of  buyers,  contract  administrators,  emd 
experience  level  of  the  work  force.  Operational  contracting 
outputs  may  include  the  total  contracting  actions,  total 
line  items  awarded,  or  total  dollars  obligated. 

A  final  major  advantage  of  the  DEA  technique  is  its 
ability  to  provide  meaningful  feedback  to  contracting 
memagers.  Each  operational  contracting  squadron  can  be 
evaluated  relative  to  the  best  performing  squadrons  across 
the  Air  Force.  Once  evaluated,  the  DEA  technique  provides 
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managers  with  information  about  reductions  in  inputs  or 
increases  in  outputs  required  to  improve  performemce. 
Specific  Problem  Statement 

Headquarters  United  States  Air  Force  (SAF/AQCO)  and  the 
Air  Force  Logistics  Meuiagement  Agency  (AFLMA/L6C)  identified 
a  need  to  develop  and  evaluate  an  alternative  method  of 
measuring  operational  contracting  performance  ( 27 ) .  The 
alternate  method  must  satisfy  evaluation  and  feedback  needs 
identified  by  operational  contracting  managers.  Because  of 
its  favor2d)le  characteristics  emd  eUsility  to  measure 
performemce  in  the  service  and  non-market  sectors  of  an 
economy,  the  DEA  technique  was  chosen  as  a  possible 
alternative  to  current  methods  of  perform2uice  evaluation. 

Research  Qb_lectlves  and  Related  Investigative  Questions 

The  overall  objective  of  this  research  was  to  design  a 
OEA  evaluation  system  that  provides  an  alternative  to 
current  methods  of  measuring  operational  contracting 
perf ormemce .  As  eua  alternative  method,  the  DEA  model  should 
be  compared  to  the  needs  identified  by  memagement.  In  order 
to  address  the  specific  problem,  the  research  was  broken 
down  into  two  main  objectives.  Each  objective  contains 
related  investigative  questions  ( IQ ) . 
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Objective  1.  Develop  a  DEA  model  for  operational 


contracting. 

IQ  1:  What  are  the  most  critical  outputs  generated 
from  the  operational  contracting  function? 

IQ  2:  What  are  the  most  critical  resources  available 
to  operational  contracting  squadrons? 

IQ  3:  What  are  the  results  of  the  application  of  the 
DEA  technique  to  operational  contracting  performance 
measurement? 

Objective  2.  Evaluate  DEA's  output  euid  potential  use 
to  managers  for  performance  evaluation. 

IQ  4:  What  information  does  the  DEA  output  provide  the 
Air  Force  operational  contracting  manager? 

IQ  5;  What  characteristics  of  a  performance 
measurement  system  are  desired  by  operational  contracting 
managers  and  how  are  they  addressed  by  DEA? 

Scope  and  Limitation  of  the  Research 

The  development  of  a  DEA  model  for  operational 
contracting  squadrons  required  the  use  of  a  multi-step 
process.  The  first  step  was  to  identify  potential  resources 
and  outputs  of  operational  contracting.  Once  identified,  a 
survey  was  developed  so  that  operational  contracting 
managers  could  identify  the  most  important  input  and  output 
measures.  This  survey  also  queried  operational  contracting 
managers  about  desired  characteristics  of  a  performance 
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measurement  and  evaluation  system.  After  identification  of 
the  critical  variables,  the  DEA  model  was  constructed. 

The  DEA  model  was  constructed  using  archival  data  from 
the  BCAS  system.  The  data  included  measures  for  all 
memagement  identified  input  and  output  variables  from  fiscal 
year  1992.  Once  the  model  was  constructed,  it  was  executed 
using  standard  simplex  linear  programming  software.  DEA 
results  were  then  compared  to  the  desired  evaluation  system 
characteristics  identified  through  the  survey  of  operational 
contracting  managers. 

The  survey  to  identify  critical  contracting  inputs  and 
outputs  was  administered  to  management  at  all  Continental 
United  States  (CONUS)  operational  contracting  squadrons. 
Management  was  limited  to  Base  Contracting  Officers  (BCOs), 
Deputy  BCOs,  and  Executive  Officers/Non-Commissioned 
Officers  (NCOs).  These  positions  represent  the  top  three 
military  and/or  civilian  managers  at  each  squadron.  Because 
a  research  goal  was  to  construct  a  model  based  upon 
management  input,  no  attempt  was  made  to  survey  non¬ 
management  personnel  concerning  critical  inputs  and  outputs. 

Once  inputs  euid  outputs  were  identified  by  CONUS 
contracting  management,  archival  data  were  collected  from  45 
operational  contracting  squadrons  within  Air  Force  Material 
Command  (AFMC)  and  Air  Combat  Command  (ACC).  These  commands 
were  selected  because  their  archival  input  and  output  data 
were  readily  availcUsle  for  use  in  this  investigation  of  DEA 
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appliceUsility.  The  input  and  output  data  from  these  45 
bases  were  used  for  DEA  model  development.  The  DEA  results 
reported  in  Chapter  IV  are  limited  to  a  comparison  of 
perform^mce  for  the  45  AFMC  and  ACC  bases. 


Operational  Definitions 

The  majority  of  operational  definitions  used  in  the 
research  are  included  in  the  body  of  the  thesis.  This 
approach  allows  the  reader  immediate  reference  to  a  specific 
definition  as  it  is  initially  presented.  However,  the 
following  terms  are  used  extensively  and  require  initial 
operational  definition. 

Inputs  -  The  resources  available  to  an  organization 
which  are  used  in  the  generation  of  outputs.  In  the 
operational  contracting  squadrons,  examples  of  inputs 
include  the  number  of  buyers,  contract  administrators, 
and  the  average  experience  level  of  the  personnel. 

Outputs  -  The  products  or  services  produced  by  an 
organization.  In  the  operational  contracting  squadron, 
examples  of  outputs  include  the  contract  administrative 
lead  time  (time  elapsed  from  receipt  of  purchase 
request  until  contract  award),  the  total  number  of 
contracts  awarded,  and  the  total  dollars  obligated. 

Performemce  -  Overall  manner  in  which  an  organization 
(the  operational  contracting  squadron)  fulfills  its 
intended  purpose  as  measured  against  some  standard. 

Productivity  -  The  relationship  between  the  output  of 
an  organization  and  the  inputs  that  have  gone  into 
producing  the  output.  Often  depicted  as  measures  such 
as  output  per  man-hour. 

Efficiency  -  A  concept  closely  related  to  productivity. 
Most  often  concerned  with  the  relationship  between 
scarce  resources  and  the  outputs  of  an  organization. 
Efficiency  is  used  as  a  criterion  for  judging  how  well 
an  organization  has  allocated  its  available  resources 
to  produce  outputs. 
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Productive  Efficiency  ■>  Because  the  concepts  of 
productivity  euid  efficiency  are  closely  related, 
performance  measurement  literature  often  use  the  terms 
together.  For  purposes  of  this  research,  all  three 
terms  refer  to  the  process  of  using  inputs  to  create 
outputs  in  the  operational  contracting  squadron. 

Decision  Making  Unit  (OMU)  -  In  the  DEA  methodology,  a 
DMU  is  the  orgemizational  element  which  is  analyzed 
relative  to  similar  elements.  For  the  purpose  of  this 
research,  a  DMU  is  an  operational  contracting  squadron. 

Relative  Efficiency  -  The  efficiency  of  a  DMU  when 
compared  to  all  other  DMUs. 

Absolute  Efficiency  -  The  efficiency  of  an  organization 
when  compared  to  some  hypothetical  standard  of 
efficiency. 

CALT  -  Contract  Administrative  Lead  Time.  CALT  is  the 
time  elapsed  from  the  moment  a  purchase  request  is 
received  in  the  contracting  squadron  to  the  time  the 
purchase  is  made. 

ACC  -  Air  Combat  Command. 

AFMC  -  Air  Force  Material  Command. 

BCAS  -  Base  Contracting  Automated  System.  The 
management  information  and  reporting  system  currently 
used  by  operational  contracting  squadrons. 

7106  Report  -  BCAS  Contracting  Activity  Report  (RCS 
HAF-RDC  M&A  7106).  This  report  provides  a  majority  of 
the  management  information  to  contracting  squadrons. 


Structure  of  Thesis 

Chapter  II  reviews  the  applicable  literature  concerning 
performance  measurement  in  operational  contracting 
squadrons.  The  chapter  is  designed  to  demonstrate  the 
importance  of  an  alternate  method  for  measuring  operational 
contracting  performance.  The  reader  is  introduced  to  the 
general  functions  of  the  operational  contracting  squadron. 
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the  importance  of  performance  measurement  in  these 
squadrons,  and  current  techniques  used  for  performance 
measurement.  The  chapter  concludes  with  a  introduction  to 
the  DEA  methodology  and  examples  of  application  in  public 
cuid  service  providing  sectors  of  the  economy. 

Chapter  III  explains  the  multi-step  methodology  used  to 
gather  and  use  availed>le  data.  The  first  part  of  Chapter 
III  provides  a  detailed  explanation  of  the  DEA  methodology. 
This  explanation  includes  a  graphical  solution  of  a  simple 
two  input,  single  output  operational  contracting  example. 

The  DEA  linear  program  is  also  developed  in  part  one  of 
Chapter  III. 

After  the  complete  description  of  the  DEA  methodology, 
part  two  of  Chapter  III  outlines  the  specific  techniques 
used  to  answer  each  of  the  five  investigative  questions. 
These  methods  include  a  discussion  of  the  survey  used  to 
identify  input  and  outputs,  methods  of  identifying 
characteristics  of  a  performance  evaluation  system,  and 
methods  to  compare  DEA  to  the  desired  characteristics. 

An  analysis  of  the  input  and  output  identification 
survey  is  made  in  the  initial  portions  of  chapter  IV.  The 
majority  of  the  chapter  is  devoted  to  the  analysis  of  the 
DEA  output  from  the  operational  contracting  application. 

This  analysis  concentrates  on  the  information  provided  to 
the  operational  contracting  manager  by  DEA  output.  In 
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addition,  a  comparison  of  DEA  characteristics  with  desired 
characteristics  of  a  performance  evaluation  system  is  made. 

Chapter  V  summarizes  the  research  and  draws  conclusions 
about  the  DEA  application  based  upon  the  analysis  conducted 
in  the  previous  chapter.  These  conclusions  concentrate  on 
the  signific2Uice  of  the  findings  and  practical  implications 
of  the  results.  Finally,  recommendations  for  future 
research  on  the  topic  are  made. 
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II.  Literature  Rs:gigw 


QYgry.isM 

The  purpose  of  this  chapter  is  to  provide  a  review  of 
the  applicable  literature  concerning  performance  measurement 
in  operational  contracting  squadrons.  The  chapter  is 
divided  into  seven  specific  areas  designed  to  demonstrate 
the  importance  of  an  alternate  method  for  measuring 
operational  contracting  performance. 

First,  the  general  functions  of  operational  contracting 
are  described  in  greater  detail.  The  second  section  covers 
the  importance  of  measuring  the  performance  of  an 
operational  contracting  squadron.  This  includes  reasons 
that  are  both  specific  to  the  DOD  and  appliceJale  to  all 
purchasing  activities.  A  working  definition  for  purchasing 
performance  is  reviewed  in  the  chapter's  third  section.  The 
fourth  part  of  the  chapter  concentrates  on  a  review  of  the 
four  methods  currently  used  to  evaluate  operational 
contracting  squadrons.  Specific  weaknesses  of  these  methods 
are  highlighted.  The  fifth  section  discusses  techniques 
that  are  often  used  to  measure  purchasing  performance 
outside  the  DOD.  Once  again,  technique  weaknesses  are 
highlighted.  The  sixth  section  provides  general 
difficulties  in  measuring  performance  in  public  agencies. 
This  section  concentrates  on  a  lack  of  quantifiable  measures 
and  the  inability  to  estimate  a  production  function. 
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The  final  pairt  of  the  chapter  introduces  an  alternate 
method  of  measuring  performance  for  possible  use  in  the 
operational  contracting  environment.  This  alternate  method 
is  defined  as  Data  Envelopment  Analysis  (DEA) .  The  DEA 
literature  is  reviewed  in  order  to  esteUt>lish  its  £d3ility  to 
address  shortcomings  of  other  avail2d>le  performance 
measurement  techniques.  Previous  applications  of  the  DEA 
technique  demonstrate  its  possible  use  in  the  measurement  of 
operational  contracting  squadron  performance. 

The  Function  of  Operational  Contracting  Squadrons 

The  operational  contracting  squadron  is  the 
organization  responsible  for  purchasing  the  majority  of 
supplies  and  services  for  each  Air  Force  Base.  These 
purchases  are  made  in  direct  support  of  Air  Force  mission 
requirements.  The  purchasing  function  may  be  as  simple  as 
office  supplies  procurement  or  as  complex  as  the 
construction  of  facilities. 

In  general,  operational  contracting  purchases  are 
classified  into  three  categories.  The  first  category  is 
services  contracting.  Examples  of  services  contracting 
include  the  award  and  administration  of  housing  maintenance, 
grounds  maintenance,  vehicle  rental,  and  laundry  service 
contracts.  Service  contracts  also  include  less  complex 
items  such  as  repair  of  computers,  office  equipment,  and 
other  organizational  support  equipment  (20:26). 
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The  second  category  of  operational  contracting 
purchases  is  construction.  Construction  purchasing  includes 
all  contracts  for  the  alteration,  repair,  and  maintenance  of 
existing  facilities.  Operational  contracting  squadrons  may 
also  award  contracts  for  construction  of  new  facilities 
valued  under  $200,000.  Finally,  construction  activities 
include  the  award  and  administration  of  contracts  with 
Architect-Engineering  firms  for  design  services  (20:27). 

The  final  category  of  operational  contracting  purchases 
is  commodities.  This  category  includes  a  variety  of 
supplies  and  equipment  (20:29).  Supplies  include  office 
items,  vehicle  parts,  medical  stores,  fuel,  and  other  items 
needed  to  support  the  base  mission.  Commodity  purchases 
also  include  items  like  computers,  typewriters,  and 
radiology  equipment. 

A  typical  contracting  squadron  consists  of  three 
flights  which  support  each  category  of  purchases.  In 
addition,  each  squadron  contains  a  fourth  flight  responsible 
for  the  Base  Contracting  Automated  System  (BCAS)  and 
memagement  information  analysis.  Finally,  squadrons  are 
managed  by  a  Squadron  Commander,  Deputy  Base  Contracting 
Officer  (BCO),  and  an  Executive  Officer/Non-Commissioned 
Officer. 

The  range  of  purchasing  activities  conducted  by  Air 
Force  Operational  Contracting  Squadrons  is  enormous.  In 
general,  the  squadron  supports  the  entire  mission  of  each 
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Air  Force  Base.  An  underst^ulding  of  these  varied 
responsibilities  euid  their  importance  to  mission  objectives 
is  critical  to  understemding  the  importemce  of  measuring 
contracting  performance . 

Importance  of  Performance  Measurement  in  Operational 

Cantcagtiaq 

As  previously  discussed,  a  primary  reason  for  measuring 
operational  contracting  performance  is  its  importemce  to 
supporting  the  mission  of  each  Air  Force  Base.  However, 
there  are  several  other  reasons  which  promote  p(«  formeuice 
measurement  of  the  operational  contracting  function.  These 
reasons  include  items  specific  to  the  DOD  and  items  which 
can  be  generalized  across  all  contracting  or  purchasing 
organizations . 

With  expected  budget  and  manpower  constraints, 
improvements  in  performance  are  critical  to  maintaining 
current  levels  of  defense.  Specifically,  each  DOD  function 
will  be  forced  to  find  productivity  improvements  to  offset 
budget  reductions  (53:10).  Contracting  squadrons  must 
accomplish  more  with  declining  resources.  Because 
operational  contracting  is  a  labor  intensive  service 
organization,  increasing  productivity  is  a  desirable  and 
achievable  goal  (59:28). 

In  an  attempt  to  meet  expected  declines  in  budgets,  the 
DOD  is  implementing  the  Defense  Business  Operations  Fund 
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(DBOF).  The  initial  implementation  of  DBOF  began  in  fiscal 
year  1992  (28:45).  Under  DBOF/  each  mission  support 
activity  will  eventually  compete  for  business  with  other 
activities  performing  like  fiinctions.  For  example,  a  Wing 
Commander  may  choose  between  different  Air  Logistic  Centers 
(ALC)  for  aircraft  maintenance.  The  Wing  Commander  will  be 
provided  unit  cost  figures  from  each  ALC  and  be  ed^le  to 
choose  the  lowest  cost  service  (53:11). 

The  cost  information  used  under  DBOF  will  be  based  upon 
unit  cost  measures.  The  prices  charged  for  service 
functions  will  equate  to  a  cost  per  unit  of  output  (53:11). 
If  a  service  providing  function  can  increase  its 
productivity  and  efficiency,  it  will  lower  its  unit  cost  emd 
become  more  competitive.  The  alignment  of  costs  to  outputs 
will  force  decision~makers  to  reduce  costs  by  changing  the 
output  or  changing  the  process  which  generates  the  output 
(53:11) . 

Although  DBOF  is  not  fully  implemented,  operational 
contracting  squadrons  may  eventually  fall  under  this 
program.  The  long  range  goal  of  DBOF  is  to  bring  all 
mission  support  activities  into  the  fund  (53:11). 

Performeuice  measurement  will  be  critical  to  improving 
productivity  amd  lowering  operational  contracting  costs. 
Lower  costs  would  allow  a  contracting  squadron  to  keep  its 
current  level,  or  even  increase  its  customer  base. 
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The  importauice  of  measuring  contracting  performemce  is 
not  restricted  to  the  DOO.  A  study  of  purchasing  managers 
by  vem  Weele  revealed  five  general  reasons  for  measuring 
purchasing  performemce.  Each  reason/  along  with  an 
appropriate  operational  contracting  example,  is  provided  in 
Table  2.1. 
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TJUUiB  2.1  Iteasoaa  For  Iteasuring  Purchasing  Parforaanca 

1.  Purchasing  performance  evaluation  ceui  lead  to 
better  decision  making.  It  identifies  strengths, 
weaknesses,  emd  variamces  from  planned  results.  These 
problems  can  be  analyzed  and  then  prevented  in  future 
operations.  In  the  operational  contracting  squadron, 
management  could  compare  measured  Contract  Administration 
Lead  Time  (CALT)  against  an  estedilished  goal. 

2.  The  measurement  of  purchasing  performamce  may  lead 
to  better  communication  with  other  departments.  An  example 
is  analyzing  payment  conditions  with  financial  meinagement 
and  deciding  on  specific  payment  procedures  to  improve 
mutual  understanding.  In  the  Air  Force,  operational 
contracting  performemce  evaluation  may  lead  to  better 
receiving  operations  in  the  Base  Supply  Squadron  or  a  more 
efficient  method  of  payment  with  Accounting  and  Finance. 

3.  Measuring  purchasing  performance  makes  things 
visible.  "The  regular  reporting  of  actual  versus  pleumed 
results  enables  a  buyer  to  verify  whether  his  or  her 
expectations  have  been  realized"  (56:18).  In  operational 
contracting  this  feedback  is  not  limited  to  the  buyer,  but 
also  provides  performance  feedback  to  all  levels  of 
management . 

4.  Performance  evaluation  may  contribute  to  better 
motivation.  A  well  designed  evaluation  system  can  be  used 
in  a  constructive  goal  setting,  motivational, 
organizational,  and  personnel  development  program.  The 
ability  for  an  operational  contracting  squadron  to  obtain  a 
performeuice  measure  can  motivate  toward  a  continuous  process 
of  improvement. 

5.  Purchasing  performance  evaluation  should  result  in 
a  higher  added  value  of  the  purchasing  function  to  the  firm. 
For  operational  contracting,  this  higher  added  value  may 
include  lower  operating  cost,  lower  service  and  supply 
costs,  better  sourcing  decisions,  and  a  myriad  of  other 
process  improvements. 

(56:18) 


After  establishing  reasons  for  measuring  performance  of 
operational  contracting  squadrons  it  is  necessary  to  review 
a  definition  of  purchasing  performance.  This  definition 
concentrates  on  concepts  of  effectiveness  and  efficiency. 
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A  Definition  of  ^chasing  Performance 


The  performance  of  ^ul  operational  contracting  squadron 
is  contingent  on  two  primary  concepts.  A  squadron  can  be 
considered  a  good  performer  if  it  is  effective  and 
efficient.  Additionally,  effectiveness  emd  efficiency  are 
not  mutually  exclusive.  "Whether  a  certain  result  should  be 
considered  as  effective  or  efficient  varies  depending  on  the 
aggregation  level  from  which  the  matter  is  perceived" 
(56:19).  For  example,  the  Chief  Executive  Officer  of  a 
compeuiy  may  perceive  lowering  the  purchasing  department 
budget  as  a  measure  of  efficiency  while  the  purchasing  agent 
may  believe  it  is  a  measure  of  effective  operations. 

Purchasing  effectiveness  is  defined  as  the  ability  to 
meet  a  previously  established  goal  or  standard. 

Effectiveness  is  a  concept  which  refers  to  the  relationship 
between  actual  ^uld  planned  performance  of  any  activity 
(56:19).  In  operational  contracting,  specific  goals  may  be 
established  at  all  levels  of  memagement. 

Purchasing  efficiency  is  the  relationship  between 
"planned  and  actual  sacrifices  made  in  order  to  realize  a 
goal  previously  agreed  upon"  (56:19).  Efficiency  is  a 
concept  closely  related  to  productivity.  Productivity  is 
often  considered  to  be  the  actual  ratio  of  outputs  produced 
by  an  organization  compared  to  the  inputs  used  to  create  the 
output  (42:420).  The  Americem  Productivity  Center  defines 
productivity  as  a  ratio  of  quantity  of  output  to  quantity  of 
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input  for  a  given  process  (47:1).  An  operational 
contracting  example  of  productivity  or  efficiency  may  be  the 
number  of  purchase  orders  awarded  per  buyer. 

This  reseeurch  concentrates  on  measuring  operational 
contracting  performeuice  as  it  relates  to  productivity  and 
efficiency.  Memy  effectiveness  goals  for  Air  Force 
operational  contracting  are  pre-ested)lished.  For  example, 
the  Federal  Acquisition  Regulation  specifies  the  use  of 
competition  to  the  meucimum  extent  possible  and  each  major 
command  esteJalishes  a  goal  for  the  use  of  competitive 
purchase  procedures.  The  operational  contracting  squadron's 
main  objective  is  to  produce  the  maximum  output  using  the 
lowest  level  of  inputs. 

After  establishing  a  general  working  definition  for 
purchasing  performance  evaluation,  it  is  necessary  to  review 
current  methods  used  to  measure  contracting  performance. 

This  review  concentrates  on  techniques  used  and  their 
corresponding  weaknesses.  The  review  is  divided  in  two 
sections.  The  first  section  describes  current  methods  used 
by  the  Air  Force  to  measure  operational  contracting 
squadrons.  The  second  section  covers  other  techniques  used 
to  measure  purchasing  performance  in  the  private  sector. 

Current  Operational  Contracting  Performance  Evaluation 

The  current  methods  of  evaluating  operational 
contracting  performance  include  four  primary  techniques. 
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These  techniques  are  the  Self-Inspection,  the  Command 
Inspector  General  (IG)  visit.  Air  Force  and  Major  Command 
Goals,  and  the  use  of  BCAS  measures.  Each  technique 
produces  advantages  and  disadvemtages  for  the  operational 
contracting  manager. 

The  Self-Inspection.  Operational  contracting  self- 
inspection  may  be  local  management's  single  most  effective 
management  tool  for  assessing  the  health  of  the  organization 
(20:101).  A  good  self-inspection  program  integrates  IG 
findings,  IG  crossfeeds,  and  command  inspection  guides  into 
a  checklist  used  by  the  operational  contracting  squadron. 

The  strength  of  a  self-inspection  rests  with  the  squadron's 
ability  to  review  its  operations  when  management  determines 
the  need. 

However,  the  self -inspection  process  has  several 
inherent  weaknesses.  First,  it  is  normally  a  compliance 
based  management  technique.  Checklists  normally  address  yes 
or  no  type  questions.  For  example,  a  self-inspection 
checklist  may  ask  whether  the  proper  documentation  to 
dissolve  a  small  business  set-aside  is  present  in  a  purchase 
order.  This  compliance  based  review  provides  little 
feedback  on  productivity  and  efficiency  in  operational 
contracting  squadrons. 

Second,  the  self -inspection  program  provides  a  limited 
ability  to  compare  squadron  performance  with  other 
contracting  squadrons.  A  self-inspection  may  include  how 
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well  the  squadron  is  performing  when  compared  to  a 
checklist.  However,  this  inspection  often  fails  to  provide 
performance  feedback  relative  to  other  squadrons. 

Finally,  the  feedback  provided  through  a  self¬ 
inspection  is  directly  related  to  the  niunber  of  man-hours 
required  to  complete  the  process  and  the  experience  level  of 
the  inspectors.  A  self -inspection  requires  removing 
manpower  from  purchasing  activities.  During  periods  of  high 
purchasing  activity  there  is  an  incentive  for  less  effort  in 
pursuit  of  a  quality  self -inspection. 

The  Command  IG  Visit.  The  IG  visit  is  a  formal 
evaluation  of  operational  contracting  performance.  The 
results  of  the  IG  inspection  become  a  matter  of  official 
record  (20:106).  Like  a  self-inspection,  the  16  review  is  a 
compliance  based  process.  The  16  performance  evaluation 
concentrates  more  on  issues  of  compliance  with  Federal 
Acquisition  Regulations  than  on  actual  indicators  of 
efficiency  and  productivity.  This  is  not  to  imply  that 
compliance  with  applicable  procurement  law  is  not  important, 
but  the  process  of  efficiently  using  available  resources  to 
produce  output  receives  little  attention  under  IG  review. 

The  IG  process  does  provide  the  operational  contracting 
squadron  with  relevant  feedback  in  a  formal  report  (20:109). 
However,  the  frequency  of  management  feedback  is  relatively 
low.  IG  inspections  usually  occur  once  every  twelve  to 
eighteen  months.  By  the  time  a  performance  problem  is 
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identified  by  the  IG,  the  contracting  squadron  may  have  been 
conducting  less  than  optimal  operations  for  quite  some  time. 

Air  Force  and  Major  Command  Goals.  Each  fiscal  year, 
the  Air  Force  ^uld  its  Major  Commands  develop  specific  goals 
to  judge  operational  contracting  perf ormemce .  These  goals 
concentrate  on  the  areas  of  competition  and  support  of 
socio-economic  objectives.  For  exeunple,  each  squadron  is 
judged  against  a  competitive  purchase  objective  and  a  small 
business  usage  standard  established  by  higher  management. 

These  goals  do  provide  some  measure  of  performance 
against  other  contracting  squadrons.  However,  the  range  of 
contracting  functions  measured  is  relatively  limited.  The 
data  measured  is  objective  in  nature,  but  these  goals 
provide  little  feedback  concerning  productivity  or 
efficiency.  The  ability  to  award  a  majority  of  purchases  to 
small  business  concerns  says  little  about  the  efficient  use 
of  resources  to  produce  outputs. 

EGAS  Measures.  The  development  of  EGAS  has  provided  a 
significant  area  of  improvement  over  previous  operational 
contracting  management  information  systems.  This  system 
provides  on  line  and  real  time  processing  of  contracting 
activity  (45:14).  Like  most  management  information  systems, 
EGAS  is  not  without  its  data  problems,  but  it  does  provide 
management  with  timely  feedback  on  operations. 

EGAS  provides  the  manager  with  a  myriad  of  individual 
performance  indicators.  For  example,  the  EGAS  Gontracting 
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Activity  Report  <HAF-RDC  M&A  7106)  provides  more  management 
data  than  any  other  single  report  in  BCAS  (45:130).  As  of  1 
October  1992,  the  7106  report  generated  one  hundred  fifty 
five  individual  indicators  of  contracting  performance. 
Examples  of  these  indicators  include  total  dollars 
obligated,  total  actions  executed,  and  total  small  business 
dollars  awarded. 

The  large  number  of  possible  measures  available  from 
BCAS  is  an  indicator  of  its  drawbacks.  There  is  no  method 
for  an  operational  contracting  manager  to  determine  overall 
performance  with  such  a  large  number  of  individual  measures. 
Additionally,  relative  comparison  between  different 
squadrons  is  not  possible  under  the  current  system. 

Additional  Techniques  Available  to  Measure  Purchasing 
Performance 

In  addition  to  the  techniques  currently  used  to  measure 
operational  contracting  performance,  two  alternate  methods 
are  available.  These  methods  are  the  use  of  ratio  analysis 
and  multiple  regression  (49:8).  Each  of  these  methods  have 
been  well  tested  in  practice  and  management  literature. 

Both  have  drawbacks  which  limit  their  use  in  the  purchasing 
environment . 

Ratio  Analysis  and  Extensions.  Ratio  analysis  and  the 
use  of  accounting  measures  of  efficiency  are  currently  used 
by  commercial  industries  to  measure  purchasing  productivity 
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(58:46).  Additionally,  limited  use  of  ratio  analysis  is 
used  by  operational  contracting  managers.  For  exeunple,  a 
comparison  may  be  made  of  the  number  of  contracting  actions 
awarded  per  purchasing  employee.  Other  examples  of  singular 
accounting  measures  include  the  ratio  of  competitive  actions 
awarded  to  the  total  actions  awarded,  the  average  Contract 
Administrative  Lead  Time  (CALT),  or  the  percentage  of  total 
dollars  awarded  to  small  business  concerns  (45:31). 

The  use  of  ratio  analysis  has  serious  shortcomings. 
First,  ratio  analysis  fails  to  account  for  differences 
between  each  organization  (49:8).  For  example,  differences 
in  contracting  squadron  mission,  size,  complexity  of 
purchases,  and  quantity  and  quality  of  human  resources  are 
difficult  to  account  for  under  ratio  analysis.  This  makes 
meaningful  comparisons  difficult  because  no  two  operational 
contracting  squadrons  are  the  same.  Second,  ratio  analysis 
is  limited  to  an  examination  of  single  combinations  of 
resources  and  outputs.  Service  providers,  such  as 
operational  contracting,  have  numerous  resources  and  outputs 
which  are  impozrtant  to  overall  performance. 

To  account  for  the  multiple  input  and  output 
characteristics  of  a  service  provider,  analysts  often 
compute  several  ratios  simultaneously  (49:9).  Recently,  Air 
Combat  Command  (ACC),  has  implemented  a  spreadsheet 
collection  of  several  individual  ratios  of  operational 
concracting  performance  (55).  Currently,  no  method  exists 
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to  combine  these  individual  ratios  into  a  single  index  of 
performance.  When  exeunined  collectively,  ratio  measures 


"present  a  morass  of  numbers  that  give  no  clear  indication 
of  true  efficiency"  or  overall  performeuice  (49:9). 

To  depict  the  problem  presented  by  ratio  euialysis,  a 
hypothetical  example  has  been  adopted  from  Sexton  for  the 
operational  contracting  environment  (49:2).  The  following 
example  of  four  operational  contracting  squadrons  will  be 
revisited  in  the  Chapter  III  discussion  of  methodology. 
Listed  in  Table  2.2  are  each  squadron's  resources  and 
outputs  generated  over  the  past  year. 


TABLE  2.2  Hypothetical  Resources  and  Outputs 


#  of  buyers 

Operating  budget 

Total  actions 

Total  dollars 

Base 

A  6 

2,000 

4,000 

15M 

Base 

B  4 

5,000 

3,000 

12M 

Base 

C  10 

8,000 

6,000 

20M 

Base 

D  10 

3,000 

10,000 

18M 

These  two  inputs  and  outputs  could  be  used  to  form  several 
areas  of  ratio  analysis.  Three  examples  are  presented  in 
Table  2.3.  Squadron  rankings  are  provided  in  parenthesis. 

TABLE  2.3  Example  Ratio  Analysis 


Actions 

per  buyer 

Dollars  per  buyer 

Dollars  spent  per 
dollar  of  operating  budget 

Base 

A 

666. 

7  (2) 

2.5M 

(2) 

7,500 

(1) 

Base 

B 

750 

(1) 

3.0M 

(1) 

2,400 

(4) 

Base 

C 

600 

(3) 

2.0M 

(3) 

2,500 

(3) 

Base 

D 

300 

(4) 

1.8M 

(4) 

6,000 

(2) 

Based  on  the  example  ratio  analysis,  it  is  impossible 
to  identify  which  hypothetical  contracting  squadron  is  most 
productive  or  efficient.  For  example,  when  analyzing 
actions  per  buyer  and  dollars  per  buyer.  Base  B  is  the  most 
efficient.  However,  Base  B  is  the  least  efficient  when 
analyzing  dollars  spent  per  dollar  of  operating  budget.  In 
the  dollars  spent  per  dollar  of  operating  budget  category. 
Base  A  is  the  most  efficient,  but  it  ranks  second  in  actions 
per  buyer  and  dollars  spent.  Which  base  is  the  most 
efficient  overall?  This  simple  example  demonstrates  the 
weaknesses  caused  by  the  inability  to  collectively  analyze 
multiple  ratios  (49:20). 

In  an  effort  to  overcome  problems  with  basic  ratio 
emalysis,  two  alternate  measurement  techniques  have  been 
used  by  purchasing  management.  First,  attempts  have  been 
made  to  benchmark  singular  performance  measures  to  provide 
meaningful  comparisons  between  purchasing  functions  within  a 
particular  industry.  The  purchasing  benchmark  project  of 
the  Center  for  Advanced  Purchasing  Studies  (CAPS)  has  begun 
to  gather  purchasing  data  from  firms  in  selected  industry 
groups.  When  the  data  is  collected  and  the  industry  group 
benchmarked,  individual  firms  can  compare  performance 
against  a  stcmdard  for  each  purchasing  performance  measure 
(39:77) . 

The  primary  drawback  to  the  benchmarking  technique  is 
identical  to  problems  with  steuidard  ratio  analysis.  A 
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meaningful  measure  of  overall  productivity  is  not  possible 
using  individual  benchmarks  for  comparison.  Attempting  to 
address  this  problem,  the  Six  Sigma  Barometer  was  developed 
by  the  Digital  Equipment  Corporation  (36:7).  The  Six  Sigma 
Barometer  attempts  to  aggregate  individual  performemce 
measures  into  a  single  index  of  overall  performance  through 
the  use  of  an  a  priori  weighting  scheme  (36:10). 

The  critical  irawback  with  this  combination  technique 
is  the  relative  importance  of  each  individual  measure  varies 
from  firm  to  firm.  In  the  operational  contracting  example 
( Table  2.3),  the  actions  per  buyer  measure  may  be  more 
important  for  Base  B  than  for  Base  D.  In  s\nnmary,  the 
required  weight  for  each  measure  will  vary  upon  the  mission 
requirements  of  the  contracting  squadron,  and  the  level  of 
management  deciding  on  appropriate  weights  (11:64). 

The  general  difficulty  of  using  combinations  of 
singular  performance  measures  is  the  requirement  for  each 
contracting  squadron  to  use  the  seune  weighting  factor  for 
each  measure.  The  weight  chosen  will  be  a  compromise  of 
perceived  importance  of  each  measure  by  the  contracting 
squadrons  and  those  performing  the  evaluation.  A  single 
weighting  scheme  is  derived  to  account  for  the  entire  range 
of  possible  functional  responsibilities,  organizational 
goals,  and  types  of  purchases.  If  these  factors  vary  from 
squadron  to  squadron,  the  resulting  performance  index  will 
inaccurately  reflect  true  purchasing  efficiency.  Finally, 
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the  ratio  analysis  technique  and  extensions  do  not  provide 
contracting  managers  information  concerning  which  areas  of 
performance  to  chemge  in  order  to  improve  overall  efficiency 
(39:8) . 

Multiple  Regression.  As  em  alternative  to  ratio 
emalysis,  multiple  regression  is  frequently  used  to  measure 
performemce  in  a  service  providing  industry.  Most  often,  a 
model  is  constructed  with  some  output  measure  as  the 
dependent  or  Y  variable  and  various  input  measures  as  the 
independent  or  X  variables  (49:9).  In  the  previous  example 
of  two  contracting  inputs  and  two  contracting  outputs,  a 
single  output  may  be  modeled  against  two  inputs.  Data  from 
all  operational  contracting  squadrons  would  be  collected  and 
used  to  create  a  predictive  equation.  For  example,  the 
total  dollars  awarded  may  be  the  dependent  variable  and  the 
operating  budget  and  number  of  buyers  may  be  the  independent 
variables  as  shown  in  Equation  ( 1 ) . 

Total  dollars  =  Po  +  pi*(oper.  budget)  +  P2*(#  of  buyers)  +  e  (1) 

In  this  example,  the  regression  technique  can  ptovide 
an  estimated  relationship  that  could  be  used  to  predict  the 
total  dollars  awarded  based  on  operating  budget  and  number 
of  buyers  (49:9).  If  a  particular  contracting  squadron  was 
relatively  efficient,  the  actual  total  dollars  awarded  would 
exceed  the  level  predicted  by  the  regression  model.  In 
mathematical  terms,  the  error  term  of  efficient  squadrons 


32 


would  be  positive.  Conversely,  a  relatively  inefficient 
squadron  would  have  a  higher  predicted  value  than  actually 
exists  and  therefore  a  negative  error  term. 

There  are  a  number  of  theoretical  drawbacks  to  the 
multiple  regression  technique.  These  theoretical  problems 
can  be  translated  into  difficulties  which  operational 
contracting  squadrons  would  face  if  the  regression  technique 
was  employed.  These  problems  are  outlined  below: 

1.  Single  equation  regression  analysis  would  require 
that  only  one  output  be  used  as  the  dependent  variable.  If 
more  than  one  output  is  desired,  the  analyst  must  combine 
multiple  outputs  into  one  measure.  In  the  hypothetical 
example,  the  total  actions  and  total  dollars  would  have  to 
be  combined  using  some  type  of  a  priori  weighting  scheme 
(49:9). 

2.  Using  the  error  term  as  the  efficiency  measure,  the 
regression  analysis  would  measure  productive  efficiency  of 
contracting  squadrons  relative  to  average  performers  as 
opposed  to  the  best  performers.  This  average  measure  would 
provide  contracting  managers  with  diluted  information 
concerning  possible  productivity  gains  (49:9). 

3.  The  use  of  regression  analysis  would  require  the 

a  priori  specification  of  a  parametric  production  function. 
The  appropriate  form  of  the  operational  contracting 
production  function  must  be  created  prior  to  evaluation. 

Are  the  inputs  combined  in  a  linear,  exponential. 
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multiplicative,  or  log  fashion  when  creating  outputs?  An 
improper  decision  in  the  model  creation  stage  would 
introduce  specification  bias  into  the  results  (25:180). 

The  specific  weaknesses  of  using  ratio  analysis, 
extensions  of  ratio  analysis,  and  multiple  regression  to 
measure  operational  contracting  performemce  arise  from  the 
nature  of  the  contracting  function.  Contracting  m£U[xagers 
have  difficulty  measuring  resource  inputs  emd  process 
outputs.  Once  measured,  the  combination  of  these  multiple 
inputs  and  outputs  into  a  single  meaningful  index  becomes 
difficult.  Which  performance  attributes  are  the  most 
critical?  How  should  differences  in  mission  objectives, 
complexity  of  purchases,  and  organizational  characteristics 
be  taken  into  account?  The  first  step  in  understanding 
these  problems  can  be  found  by  examining  performance 
measurement  difficulties  in  public  agencies  and  problems 
associated  with  measuring  service  orgeinization  perf oinnance . 

Performance  Measurement  Difficulties  in  Public  Agencies 

The  difficulties  of  measuring  productivity  or 
efficiency  in  the  public  sector  have  been  well  documented. 
The  general  problem  can  be  classified  into  several  specific 
areas  related  to  a  lack  of  quantifi6d:>le  measures  and 
difficulty  in  estimating  a  public  agency  production 
function.  The  same  measurement  problems  identified  in 
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previous  literature  are  applicable  to  the  public  agency 
function  of  operational  contracting. 

Lack  of  Quantifiable  Measures.  The  first  problea  of 
measuring  public  agency  performance  is  the  general 
difficulty  in  determining  profit.  Some  argue  public  sector 
managers/  because  they  do  not  make  profits,  are  less 
efficient  than  private  sector  counterparts  (46:74). 
Independent  of  this  argument  is  the  fact  that  a  measure  of 
profit  is  the  key  factor  in  establishing  effectiveness 
criteria  in  private  organizations  (3:40).  In  a  profit- 
oriented  firm,  management  decisions  and  performance 
evaluations  are  based  on  the  attainment  of  profit. 

Military  orgemizations,  including  the  operational 
contracting  squadrons,  are  service-oriented  operations. 
Profit  is  not  the  primary  objective  and  management  decisions 
are  often  made  on  the  criteria  of  providing  the  best 
possible  service  with  available  resources  (3:35).  Current 
operational  contracting  performance  can  not  be  linked  to 
profit  because  it  operates  as  a  public  agency  service 
provider. 

Without  a  relevant  measure  of  profit,  public  agencies 
may  look  for  alternative  indicators  of  perf ormeuice .  Because 
these  agencies  usually  produce  a  service,  it  is  difficult  to 
quemtify  the  output  produced  by  sector  enterprises.  How 
does  an  analyst  quantify  the  amount  of  education  produced  by 
a  school  (46:75)?  How  can  operational  contracting  managers 
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measure  the  level  of  service  provided  to  their  respective 
base?  Examples  of  piiblic  sector  emd  service  providing 
functions  are  provided  below: 

1.  A  1992  study  by  Murphy  demonstrated  measurement 
problems  present  in  the  petroleum  industry  purchasing 
function.  Results  demonstrated  a  lack  of  understanding 
concerning  the  relationships  of  various  purchasing 
performemce  variables,  both  to  each  other  emd  overall 
purchasing  performance  (39:79). 

2.  A  1984  study  by  V6ui  Weele  identified  three  specific 
problems  in  measuring  emd  evaluating  purchasing  perf omumce . 
First,  there  exists  a  lack  of  formal  objectives  and 
performance  standards  for  purchasing.  Second,  purchasing  is 
not  an  isolated  function,  resulting  in  difficulty 
identifying  direct  input-output  relationships  of  the 
process.  Third,  there  exists  differences  in  the  scope  of 
purchasing.  Purchasing  tasks  and  responsibilities  may 
differ  between  different  orgcmizations  (56:18). 

3.  Memy  public  sector  organizations  have  problems 
defining  programs  and  complex  tasks  to  perform.  For 
exeunple,  public  health  agencies  have  a  hard  time  specifying 
the  correct  type  and  amount  of  therapy  to  apply  to  a 
patients  symptoms  (4:194). 

4.  A  1991  study  by  Ray  recognized  the  fact  that  inputs 
and  outputs  of  public  schools  included  items  which  contained 
limited  amounts  of  economic  meaning.  Input  measures 
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included  classroom  teachers  per  pupil,  support  staff  per 
pupil,  and  administrative  staff  per  pupil.  Output  measures 
included  student  scores  on  proficiency  exams  (44:1625). 

5.  When  examining  the  efficiency  of  highway 
maintenance  patrols.  Cook  et  al.  addressed  the  need  to 
develop  a  measurement  technique  caped>le  of  hemdling  non¬ 
economic  factors.  These  factors  included  average  age  of 
pavement,  number  of  accidents,  and  traffic  volume  per  day 
(17:114) . 

Difficulty  Estimating  a  Service  Sector  Production 

Function.  The  previously  examined  problems,  caused  by  lack 

of  quantifi2d)le  measures,  relates  to  a  larger  theoretical 

problem  when  measuring  public  and  service  sector 

organizations.  In  economic  terms,  it  is  difficult  to 

estimate  a  production  function  or  appropriate  production 

possibilities  frontier  in  public  organizations. 

Methods  for  evaluating  the  relative  productivity  of 
units  in  the  public  sector  have  lagged  behind  similar 
applications  where  production  functions  were  more 
directly  obtainable.  (7:57) 

The  difficulty  in  establishing  an  appropriate  production 
function,  coupled  with  the  measurement  problems  previously 
identified,  lead  researchers  to  look  for  an  alternative 
technique  for  measuring  performance. 

Traditional  economic  theory  defines  a  production 
function  as  the  relationship  between  inputs  and  outputs 
where  the  quantity  produced  is  equal  to  some  function  of 
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various  inputs.  In  the  simple  model,  inputs  would  include  a 
combination  of  labor  cuid  capital  (40:235).  In  the 
contracting  squadron,  labor  may  include  the  number  of 
buyers,  capital  may  include  the  availsdsility  of  computer 
usage,  and  an  output  may  be  the  number  of  contracts  awarded. 

Microeconomic  theory  defines  a  production  possibilities 
frontier  as  the  alternative  combinations  of  outputs  that  can 
be  efficiently  produced  by  a  firm  with  a  fixed  quantity  of 
resources  (40:623).  In  the  simple  case  of  two  outputs,  a 
production  possibilities  frontier  would  take  the  shape  of  a 
convex  curve  as  shown  in  Figure  2.1. 


FIGURE  2.1  Production  Possibilities  Frontier 


Economic  theory  suggests  that  any  point  along  the  production 
possibilities  frontier  is  obtaincUsle.  The  firm  should 
produce  at  some  point  along  the  curve  at  all  times  unless  it 
fails  to  allocate  its  resources  in  an  efficient  manner 
(54:213) . 
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The  difficulty  arising  from  the  use  of  the  traditional 
production  function  and  production  possibilities  frontier 
arises  when  the  model  is  extended  to  the  multiple  input  and 
output  scenario.  When  the  basic  model  is  extended,  accurate 
efficiency  measurement  can  only  be  made  if  the  actual 
production  function  is  lcno%m  prior  to  «malysis  (48:7).  In 
the  operational  contracting  example,  a  priori  decisions  must 
be  made  concerning  the  mathematical  relationship  between 
inputs  and  outputs. 

The  multiple  input,  multiple  output  problem  was  first 

addressed  by  M.J.  Farrell  in  1957  (23:253).  The  renewed 

interest  in  developing  a  working  model  to  measure  the 

production  possibilities  or  efficient  frontier  was  best 

stated  in  Farrell's  original  work: 

The  problem  of  measuring  the  productive  efficiency  of 
an  industiry  is  important  to  both  the  economic  theorist 
euid  the  economic  policy  maker.  If  the  theoretical 
argiunents  to  the  relative  efficiency  of  different 
economic  systems  are  to  be  svibjected  to  empirical 
testing,  it  is  essential  to  be  able  to  make  some  actual 
measurements  of  efficiency.  Equally,  if  economic 
plemning  is  to  concern  itself  with  particular 
industries,  it  is  important  to  know  how  far  a  given 
industry  can  be  expected  to  increase  its  output  by 
simply  increasing  its  efficiency,  without  ed^sorbing 
further  resources.  (23:253) 

This  statement  by  Farrell  is  critical  for  the  operational 
contracting  squadron.  Contracting  managers  must  be 
concerned  with  their  performance  relative  to  other 
squadrons.  Additionally,  decreasing  DOD  budgets  will  force 
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contracting  squadrons  to  increase  or  maintain  output  by  pure 
increases  in  efficiency. 

Charnes,  Cooper,  and  Rhodes  expemded  the  individual 
firm  evaluations  of  Farrell  to  a  multi-input,  multi-output 
model  for  performance  evaluation  of  multiple  orgemizations 
with  like  functions  (13:430).  A  linear  programming 
technique  to  estimate  the  efficient  frontier  was  termed  Data 
Envelopment  Analysis  (OEA)  (6:127). 

PEA  Overview 

The  specific  DEA  methodology  is  outlined  in  Chapter 
III.  The  purpose  of  this  section  is  to  introduce  the  reader 
to  the  basic  concepts  of  DEA.  This  overview  includes  bahic 
characteristics  of  DEA  which  make  it  an  appropriate  model  to 
apply  to  the  operational  contracting  environment.  Finally, 
a  discussion  of  previous  DEA  applications  aids  in 
establishing  its  usefulness  for  performance  measurement  in 
public  and  service  providing  organizations. 

DEA  is  a  mathematical  programming  technique  used  to 
measure  productive  efficiency,  or  productivity  of  each 
member  of  a  set  of  comparable  producing  units-.  These  units 
are  termed  Decision  Making  Units  or  DMUs.  DEA  measures 
productivity  in  a  relative  sense,  that  is  each  DMU  is 
evaluated  relative  to  its  peers  (49:7).  The  DEA  technique 
converts  multiple  input  euid  output  measures  into  a  single 
comprehensive  measure  of  productive  efficiency  (22:90).  The 
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ability  of  DEA  to  incorporate  several  traditional  ratio 
measures  into  a  single  performance  measure  makes  it  well 
suited  for  operational  contracting  application.  In  general, 
the  DEA  technique  floats  a  piece-wise  linear  frontier  to 
rest  on  the  top  of  observations  of  DMU  performance  (48:8). 
This  frontier  is  analogous  to  the  efficient  frontier  in 
traditional  economic  analysis  (Figure  2.1). 

The  specific  characteristics  of  DEA  make  it  an 
attractive  alternative  for  measuring  performance  in  public 
sector  and  service  providing  organizations  (34:361).  These 
characteristics,  along  with  appropriate  operational 
contracting  examples,  are  provided  in  Table  2.4. 
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TABLE  2.4  Characteristics  of  OEA 


j  1.  Through  DEA  use,  siii^llar  public  sector  DMUs  such  as 
schools  and  hospitals  may  be  compared  directly  and  evaluated 
in  terms  of  relative  efficiency  even  when  they  produce 
multiple  outputs  from  multiple  inputs. 

2.  DEA  is  appropriate  for  use  when  economic  values  for 
outputs  and  inputs  are  not  easily  identified.  For  example, 
it  is  difficult  to  place  an  economic  price  upon  the  number 
of  contracts  awarded  or  the  level  of  competition  obtained  by 
an  operational  contracting  squadron. 

3.  DEA  is  an  extremal  method  for  measuring 
performance.  The  process  compares  a  DMU  with  the  best 
performing  DMUs,  not  a  hypothesized  or  average  performer. 

4.  Each  DMU  is  evaluated  based  on  what  it  does  best. 
The  resources  suid  outputs  which  a  DMU  uses  most  efficiently 
will  receive  the  highest  possible  weight  from  the  linear 
programming  solution.  As  contracting  missions,  functions, 
and  purchase  complexities  vary  between  squadrons,  DEA  allows 
the  assigned  input  and  output  weights  to  vary. 

5.  DEA  does  not  require  specification  of  the 
importance,  rank,  or  weight  for  either  inputs  or  outputs. 
This  characteristic  eliminates  the  subjective  conflict  which 
often  arises  over  the  value  of  public  sector  resources  and 
outputs . 

6.  A  DMU  which  is  rated  relatively  inefficient  is 
indeed  strictly  inefficient.  A  DMU  is  only  inefficient  if 
another  DMU,  or  combination  of  DMUs,  produces  output  at  a 
lower  unit  cost. 

7.  Application  of  DEA,  using  controllable  inputs  or 
outputs,  allows  for  the  development  of  a  management  strategy 
for  improvement.  The  results  of  DEA  provide  information 
regarding  output  increases  or  input  reductions  needed  to 
achieve  efficiency. 

8.  DEA  input  and- output  data  can  be  used  in  raw  form. 

In  other  words,  there  is  no  need  to  standardize  data  based 
upon  units  of  measure. 

(34:361-367) 


The  characteristics  of  the  DEA  technique  make  it  a 
possible  alternative  to  measuring  performance  in  public  and 
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private  service  providing  organizations.  DEA  has  been 
tested  empirically  in  m^uly  settings.  DEA  has  been  used  to 
measure  efficiency  in  hospitals  (52;  57:185-205;  4:192-205), 
international  logistics  (29:3-14),  education  (1:165-185; 
7:57-75;  44:1620-1628),  and  banking  (24:229-245). 

Recently,  DEA  has  gained  interest  in  both  the  military 
and  purchasing  environments.  A  1992  study  by  Clarke 
evaluated  the  productivity  of  seventeen  vehicle  maintenance 
operations  in  Tactical  Air  Command  (14:376).  A  1986  study 
by  Coyle  attempted  to  apply  the  DEA  technique  to  evaluation 
of  the  Air  Force  Environmental  Program  (18:3).  Finally,  a 
1992  study  by  Murphy  demonstrated  DEA  possibilities  for 
capturing  performance  of  a  purchasing  organization.  In  this 
study,  DEA  was  applied  to  benchmarking  type  data  from  the 
purchasing  functions  in  the  petroleiun  industry.  The  DEA 
technique  proved  to  be  a  more  powerful  tool  than 
benchmarking  or  other  existing  methods  to  aggregate  the  data 
into  a  single  index  of  overall  performance  (39:209). 

Until  now,  the  DEA  technique  has  not  been  applied  to 
Air  Force  operational  contracting  squadrons.  The  importance 
of  tracking  operational  contracting  performance,  weaknesses 
in  current  methods  of  measurement,  and  the  favorable 
characteristics  of  DEA  make  its  use  a  logical  addition  to 
the  contracting  management  function.  The  many  examples  of 
DEA  in  public  and  service  providing  organizations  provide  a 
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solid  foundation  for  applying  the  technique  to  operational 
contracting. 

Chapter 

The  literature  review  has  provided  eui  overview  of  the 
operational  contracting  function.  The  operational 
contracting  squadron's  primary  objective  is  to  procure 
supplies  and  services  in  support  of  mission  requirements.  A 
general  working  definition  of  purchasing  performance  was 
provided.  This  performance  consists  of  the  related  concepts 
of  effectiveness  and  efficiency.  In  the  era  of  decreasing 
DOD  budgets,  managers  must  concentrate  on  efficiency 
improvements.  Performance  evaluation  techniques  currently 
availcdsle  for  use  in  operational  contracting  squadrons  were 
discussed.  Weaknesses  of  these  approaches  were  highlighted 
along  with  general  measurement  difficulties  present  in 
public  and  service  providing  agencies.  Finally,  the  DEA 
technique  was  introduced  as  a  possible  improvement  for 
evaluation  of  operational  contracting  squadrons. 

The  results  of  the  literature  review  indicated 
weaknesses  in  current  methods  to  measure  operational 
contracting  performance.  Current  methods  fail  to  account 
for  differences  in  squadron  characteristics,  missions,  and 
purchase  complexity.  Additionally,  contracting  managers  can 
not  combine  singular  performance  measures  into  an  aggregate 
measure  of  overall  performance.  Finally,  feedback  provided 
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by  current  methods  does  not  provide  specific  information  on 
how  to  improve  productivity  and  efficiency. 

The  recently  developed  DEA  technique  has  the  potential 
to  improve  on  current  methods  of  measuring  operational 
contracting  performance.  As  discussed  in  this  literature 
review,  the  characteristics  of  DEA  make  it  well-suited  for 
operational  contracting  perform2uice  measurement. 
Specifically,  DEA  has  the  ability  to  measure  performance  in 
service  providing  organizations  where  it  is  difficult  to 
place  an  economic  price  on  inputs  and  outputs. 
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Overview 

The  purpose  of  this  chapter  is  to  provide  an 
explanation  of  the  research  design  emd  related  OEA 
methodology.  Part  I  of  this  chapter  provides  a  detailed 
description  of  the  DEA  methodology.  Part  I  includes  ^ul 
introduction  with  DEA  assumptions,  a  simple  graphical 
excunple,  and  the  development  of  the  DEA  linear  program. 
This  description  is  required  before  the  various  techniques 
used  to  answer  specific  research  objectives  and  questions 
are  discussed. 

Part  II  of  the  chapter  provides  the  various  methods 
employed  to  answer  each  research  objective.  Specific 
techniques  for  answering  each  of  the  five  investigative 
questions  are  contained  in  this  part. 

I:  Data  Envelopment  Analysis 

In  an  effort  to  overcv^me  the  drawbacks  to  traditional 
measures  of  productivity,  outlined  in  Chapter  II,  Data 
Envelopment  Analysis  was  chosen  for  examination  as  a 
possible  alternative  for  measuring  performance  in  the 
operational  contracting  environment.  This  section  of  the 
chapter  introduces  the  reader  to  the  concept  of  DEA  and 
provides  a  detailed  explanation  of  its  application  to 
operational  contracting. 
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Introduction  to  PEA.  Since  DBA  is  the  primary 


methodology  employed  in  this  research,  a  detailed 
explanation  is  provided  as  a  first  step  in  addressing  the 
research  problem  of  developing  2m  alternate  method  of 
measuring  operational  contracting  performance.  The  same 
hypothetical  example  of  four  contracting  squadrons, 
introduced  in  the  Chapter  II  review  of  common  performemce 
measures,  is  used  to  introduce  DBA.  Bach  squadron  has  two 
inputs  emd  one  output  as  shown  in  Table  3.1. 


TABLE  3.1  Hypothetical  Squadron  Data 


#  of  buyers 

Operating  budget 

Total  dollars 

Base  A 

6 

2,000 

15M 

Base  B 

4 

5,000 

12M 

Base  C 

10 

8,000 

20M 

Base  0 

10 

3,000 

18M 

The  basic  problem  facing  the  analyst  is  to  measure  the 
relative  productive  efficiency,  or  productivity  of  each  of 
the  contracting  squadrons.  DBA  would  identify  these 
squadrons  as  a  set  of  comparable  producing  units.  The  term 
Decision  MeUcing  Unit  (DMU)  is  used  to  describe  each 
contracting  squadron  (49:7). 

DBA  is  a  mathematical  programming  technique  used  to 
measure  relative  productive  efficiency,  or  productivity  of 
each  of  a  number  of  DMUs  (39:103).  The  technique  is  well*- 
suited  for  situations  where  there  are  multiple  inputs  and 
outputs  to  the  process  and  there  is  no  way  to  aggregate 
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these  inputs  euid  outputs  into  a  meaningful  single 
perfoxmance  index  (49:10).  Difficulties  in  aggregating 
inputs  and  outputs  are  inherent  to  the  operational 
contracting  function. 

In  the  general  sense,  DEA  is  a  linear  programming 
method  of  estimating  a  production  possibilities  or  isoquant 
frontier  for  a  given  set  of  DMUs  (13:435).  The  technique 
"floats  SL  piecewise  linear  surface  to  rest  on  the  top  of 
given  DMU  observations"  (48:8).  This  surface  defines  the 
efficiency  frontier  from  which  comparisons  can  be  made  to 
determine  the  relative  efficiency  for  each  DMU  (31:428).  It 
is  important  to  note  that  efficiency  is  determined  relative 
to  other  DMUs  emd  is  not  efficiency  in  the  edssolute  sense. 

In  the  contracting  example,  each  of  the  four  squadrons  will 
be  compared  to  the  other  squadrons.  DEA  m£dces  no  attempt  to 
quemtify  the  squadrons  ed)solute  measure  of  productive 
efficiency. 

As  with  any  linear  programming  technique,  DEA  has  a  set 
of  assumptions  required  for  usage.  These  are  extensions  of 
standard  linear  programming  assumptions.  Before,  proceeding 
with  the  DEA  formulation,  these  assumptions  must  be 
addressed.  A  complete  list  of  the  DEA  assumptions,  with 
relevemt  operational  contracting  exeunples,  is  provided  in 
Ted>le  3.2. 
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TABLE  3.2  DBA  Assiunptions 


1.  DBA  requires  all  relevant  inputs  and  outputs  be  identified  and 
measured  in  a  consistent  manner  amonq  all  DMUs  (49:27) .  In  the  operational 
contracting  environment,  inputs  such  as  total  dollars  spent,  must  be  measured 
the  same  way  at  different  squadrons. 

2.  The  inclusion  of  invalid  inputs  and  outputs  can  cause  a  higher 
rating  for  some  DMUs  than  they  should  receive.  As  variables  are  added,  even 
if  irrelevant,  efficiencies  will  rise  (39:140).  This  fact  requires  an 
appropriate  method  to  identify  relevant  input  and  output  variables.  The 
method  of  choosing  variables  for  the  operational  contracting  analysis  is 
discussed  in  Part  II  of  this  chapter. 

3.  As  the  number  of  DMUs  (n)  decrease,  relative  to  the  number  of 
inputs  (m)  and  outputs  (a),  the  number  of  relatively  efficient  DMUs  increases. 
This  arises  because  DBA  solves  a  linear  program  with  (n-fl)  constraints  in 
(KH-s)  dimensional  space.  When  (m-«-s)  is  large  relative  to  n,  it  is  lilcely  that 
many  DMUs  will  find  their  optimum  solution  along  the  boundary  of  their  own 
constraint,  since  there  are  few  other  constraints  to  make  the  solution 
feasible  (39:141). 

4.  DBA  assumes  that  each  unit  of  a  given  input  or  output  is  identical 
to  all  other  units  of  the  same  type  among  DMUs(49:28). 

5.  In  working  with  a  piecewise  linear  frontier,  it  rs  assumed  that  all 
potential  points  along  the  frontier  are  feasible.  In  other  words,  it  is 
assumed  that  there  is  continuous  substitutability  of  one  input  or  output  for 
another,  such  that  every  point  on  the  line  segment  of  the  efficient  frontier 
could  be  achieved  (22:113). 

6.  DBA  addresses  the  concept  of  relative  technical  efficiency  only.  A 
DMU  IS  technically  efficient  if  it  maximizes  output  for  a  given  level  of 
inputs  (39:144).  In  the  operational  contracting  example,  each  squadron  is 
compared  relative  to  the  best  performing  squadrons  and  not  a  pre-established 
concept  of  absolute  efficiency. 

7.  DBA  assumes  there  are  constant  returns  to  scale.  Proportional 
changes  made  in  all  the  input  levels  will  result  in  changes  of  equal 
proportions  in  the  output  level.  In  other  words,  the  weights  DBA  generates 
for  inputs  and  outputs  are  constant  over  the  range  of  possible  alternatives 
(49:28) . 

8.  DBA  observes  relative  efficiency  only.  If  all  DMUs  are  inefficient 
in  an  absolute  sense,  DBA  will  not  capture  the  inefficiencies  (39:146). 

9.  Input  and  output  weights  produced  by  DBA  can  not  be  interpreted  as 
values  in  the  economic  sense.  DBA  does  not  create  economic  unit  values  for 
inputs  and  outputs,  they  are  only  used  to  measure  relative  efficiency  (49:28) . 
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These  assumptions  provide  some  DEA  limitations,  but  one 
powerful  characteristic  of  the  technique  is  its  ability  to 
examine  several  inputs  emd  outputs  simultaneously.  In  the 
operational  contracting  enviroiment,  various  inputs  and 
outputs  could  be  included  in  the  DBA  analysis.  Exeuaples  of 
possible  outputs  include  total  number  of  items  purchased  or 
contract  administrative  lead  time  (CALT).  Possible  input 
exeunples  include  the  nximber  of  purchasing  personnel  or 
average  personnel  experience.  With  this  characteristic  in 
mind,  the  first  step  in  understanding  DEA  is  to  examine  a 
two  input,  single  output  example.  This  simplified  example 
aids  DEA  understanding  by  allowing  for  a  graphical  solution. 

A  Graphical  Example.  Using  the  hypothetical  example 
previously  introduced  (Table  3.1),  a  graphical  approach  can 
be  used  to  explain  DEA.  Each  contracting  squadron  is  now 
identified  as  a  DMU.  The  single  output  is  the  total  dollars 
awarded  (in  millions).  The  two  inputs  are  the  operating 
budget  (in  thousands)  and  the  number  of  buyers.  Each  DMU 
takes  the  inputs,  and  through  the  contracting  process, 
generates  an  output.  The  process  is  shown  in  Figure  3.1. 


50 


FIGURE  3.1  Operational  Contracting  Process 


The  DEA  technique  is  employed  in  the  s£une  manner  as 
traditional  linear  programming  methods.  The  emalyst  can 
maximize  outputs  for  a  given  input  level,  or  minimize  inputs 
used  for  a  given  output  level.  In  the  graphical  ex^unple, 
inputs  are  minimized  for  a  given  level  of  output.  The  first 
step  is  to  normalize  the  data  so  the  inputs  are  expressed  in 
terms  of  a  single  unit  of  output.  The  result  of  normalizing 
the  inputs  is  shown  in  Table  3.3. 


TABLE  3.3  Normalized  Input  Levels 


DMU 

input  1/output 
#  of  buyers/total  dollars 

input  2/output 

operatinq  budqet/total  dollars 

A 

6/15  =  .40 

2/15  =  .13 

B 

4/12  =  .33 

5/12  =  .42 

C 

10/20  =  .50 

8/20  =  .40 

D 

10/18  =  .56 

3/18  =  .17 

Once  the  normalized  input  values  are  obtained,  the 
results  can  be  presented  in  graphical  form.  The  result  of 
plotting  the  normalized  inputs  per  single  output  is  an 
efficient  frontier  for  the  contracting  DMUs  (49:13).  The 
plot  of  the  normalized  input  levels  is  shown  in  Figure  3.2. 


Determining  the  Efficient  Frontier.  When  the  DBA  goal 
is  to  minimize  inputs  for  a  given  level  of  output,  any  DMU 
that  is  lower  and  to  the  left  of  another  DMU  is  more 
efficient.  This  results  because  the  efficient  DMU  is 
producing  the  same  level  of  output  using  lower  levels  of 
input.  In  the  minimization  problem,  the  ultimate  goal  of 
each  DMU  is  to  move  as  close  to  the  origin  as  possible 
(49:13).  In  the  hypothetical  example,  DMU  A  and  DMU  B  form 
the  efficient  frontier.  They  will  receive  the  maximum 
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efficiency  rating  of  1.0.  Once  the  frontier  is  formed,  all 
other  DMUs  can  be  examined  relative  to  their  location  away 
from  the  frontier.  A  graph  of  the  complete  DEA  efficient 
frontier  is  provided  in  Figure  3.3. 


RTcaminlnq  Inefficient  DMUs.  After  identifying  DMU  A 
and  DMU  B  as  the  squadrons  which  form  the  efficient  frontier 
(the  line  segment  joining  DMU  A  and  DMU  B),  an  exeuaination 
of  inefficient  DMUs  must  be  made.  This  examination  includes 
DMU  C  euid  DMU  D  from  the  graphical  example. 

DMU  C.  In  order  to  examine  the  efficiency  rating 
of  DMU  C,  a  line  segment  is  constructed  from  the  origin  so 
it  passes  through  the  efficient  frontier  and  connects  to  DMU 
C.  The  point  C*  indicates  where  the  line  segment  crosses 
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the  efficient  frontier.  To  determine  the  relative 
efficiency  of  DMU  C,  the  length  of  line  segment  OC  is 
compared  to  the  length  of  line  segment  OC.  When  OC  is 
divided  by  OC,  a  ratio  less  than  one  results  (49:15).  This 
ratio  indicates  the  relative  efficiency  of  DMU  C.  For 
example,  the  ratio  of  OC'/OC  equals  .726.  It  can  be  said 
DMU  C  receives  an  efficiency  rating  of  72.6%. 

The  information  provided  by  the  graphical  exeunple  is 
not  limited  to  a  raw  efficiency  score.  The  point  C 
corresponds  to  a  '’hypothetical  DMU  whose  input  mixture  ratio 
is  the  same  as  that  of  DMU  C  (49:15).  In  the  exeunple,  the 
hypothetical  DMU  C  uses  the  same  mix  of  buyers  and 
operating  budget  as  the  inefficient  DMU  C.  In  a  broader 
sense,  DMU  C  and  the  hypothetical  DMU  C  use  the  same 
production  technology.  This  concept  allows  DMU  C  to  use  DMU 
C  as  a  model  for  improvement. 

The  hypothetical  DMU  C  is  a  combination  of  the  two 
efficient  DMUs  which  dominate  DMU  C.  In  this  example,  DMU  A 
emd  DMU  B  dominate  DMU  C.  DMU  C*  is  a  weighted  average  of 
DMU  A  euid  DMU*  B.  In  DEA  terminology,  it  is  said  DMU  A  and 
DMU  B  form  the  efficiency  reference  set  for  DMU  C  (49:15). 
Only  efficient  DMUs  can  form  the  efficient  reference  set  for 
an  inefficient  DMU. 

DMU  D.  The  analysis  of  DMU  D  is  not  as  simple  as 
that  for  DMU  C.  DMU  D  does  not  lie  on  the  efficient 
frontier.  Moreover,  DMU  D  is  not  dominated  by  a  linear 
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combination  of  the  two  DMUs  on  the  efficient  frontier 
(8:787).  In  the  graphical  example,  a  line  can  not  be  drawn 
from  the  origin  to  DMU  D  which  passes  through  the  efficient 
frontier.  The  occurrence  of  a  point  such  as  DMU  D  is 
referred  to  as  an  "unenveloped  solution"  (8:788).  When  an 
unenveloped  solution  occurs,  the  researcher  must  proceed 
with  caution.  The  basic  DEA  will  identify  DMU  D  as  an 
efficient  DMU.  However  DMU  D  could  have  received  the  Scune 
rating  while  using  fewer  inputs  (8:789).  To  combat  this 
problem,  three  techniques  can  be  employed. 

First,  the  efficient  frontier  could  be  extended 
vertically  from  the  uppermost  DMU  and  horizontally  from  the 
lowermost  DMU.  These  extensions  correspond  to  points  B*  and 
A'  on  the  graphical  example.  These  extensions  serve  to 
provide  a  complete  efficient  frontier  for  analysis. 

However,  DMU  D  is  not  dominated  by  two  DMUs  and  a  second  DMU 
must  be  created.  In  the  graphical  example,  classical  DEA  of 
DMU  D  involves  the  creation  of  an  arbitrary  DMU  along  the 
horizontally  extended  frontier.  In  this  example,  it  would 
be  point  Z  euid  the  efficiency  rating  for  DMU  D  would  be  the 
ratio  of  line  segment  OZ  divided  by  segment  OD  (8:789). 

The  second  method  of  examining  unenveloped  solutions 
involves  the  same  concepts  as  vertical  and  horizontal 
frontier  extension.  When  the  frontier  is  extended 
vertically  and  horizontally  from  the  uppermost  and  lowermost 
DMU,  it  is  possible  to  have  a  DMU  at  point  B'  or  point  A' 
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which  receives  an  efficiency  score  of  1.0.  The  horizontal 
and  vertical  frontier  extension  would  classify  these  new 
DMUs  as  efficient,  even  though  the  new  DMUs  would  be  using 
more  of  an  input,  while  producing  the  same  output  as  DMU  B 
or  DMU  A.  Because  of  a  greater  use  of  an  input,  these  types 
of  DMUs  should  not  be  considered  efficient  (49:14). 

This  problem  can  be  avoided  by  a  redefinition  of  the 
efficient  frontier.  Instead  of  using  a  vertical  and 
horizontal  extension,  the  frontier  can  be  extended  to  points 
on  the  vertical  and  horizontal  axis  at  infinity.  This 
extension  results  in  the  highest  segment  of  the  efficient 
frontier  sloping  slightly  to  the  left.  Point  B*  would  no 
longer  fall  on  the  frontier.  Using  the  same  technique,  the 
lowest  segment  of  the  frontier  slopes  slightly  downward. 
Point  A'  is  no  longer  classified  efficient  (49:16). 

The  third  approach  to  solving  for  unenveloped  solutions 
is  the  recently  developed  extension  of  DEA  called 
Constrained  Facet  Analysis  (CFA).  CFA  was  originally 
proposed  in  1984  by  Bessent,  Bessent,  Elam,  and  Clark 
(8:785).  The  technique  attempts  to  provide  a  lower  bound 
efficiency  measure  for  DMUs  that  have  a  mix  of  resources 
and/or  outputs  which  differ  from  frontier  points  (8:785). 

CFA  extends  the  frontier  through  a  mathematical  algorithm 
which  attempts  to  envelop  all  DMUs. 

Method  Chosen  to  Analyze  Unenveloped  Solutions.  The 
proposed  CFA  extension  will  not  be  used  in  the  current  DEA 
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formulation.  Instead,  if  an  unenveloped  solution  is 
encountered,  it  will  be  identified  by  examining  the 
efficiency  reference  set  (ERS)  of  all  DMUs.  Only 
operational  contracting  squadrons  on  the  efficient  frontier 
can  appear  in  the  ERS  of  inefficient  squadrons.  Contracting 
squadrons  which  are  rated  efficient,  yet  do  not  appear  in 
the  ERS  of  another  squadron,  may  have  a  slack  value  in  one 
or  more  of  the  inputs  (39:182).  If  a  slack  value  is 
discovered,  the  contracting  squadron  could  have  produced  the 
same  level  of  output  with  lower  levels  of  identified  slack 
inputs.  In  other  words,  those  squadrons  rated  efficient, 
but  not  in  the  ERS  of  another  squadron,  and  containing  slack 
in  one  or  more  inputs,  should  not  be  considered  efficient 
(39:182).  An  examination  of  all  ERS  and  associated  slack 
values  will  identify  squadrons  which  could  have  achieved  the 
efficiency  rating  of  1.0  while  using  fewer  inputs. 

This  approach  is  analogous  to  the  graphical  example  of 
extending  the  efficient  frontier  to  points  on  the  vertical 
and  horizontal  axis  at  infinity.  This  analogy  results  from 
the  ability  to  identify  unenveloped  solutions  by  examining 
the  ERS  of  all  DMUs.  The  use  of  this  technique  increases 
the  quality  of  information  provided  from  the  DEA  model. 

The  PEA  Formulation 

The  previous  section  demonstrated  the  DEA  concept  using 
a  simple  two-input,  single-output  example.  The  graphical 
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ezeuaple  provides  a  basic  understanding  of  the  methodology, 
but  it  can  not  be  extended  to  a  multiple-input,  multiple- 
output  scenario.  Fortimately,  the  model  can  be  formulated 
into  analytical  technique  which  cem  be  solved  using  most 
linear  programming  software  packages.  This  section  explains 
the  linear  programming  formulation  for  the  DEA  methodology. 

In  order  to  understand  the  model  formulation,  the 
decision  variedsles  and  constants  require  description. 

Instead  of  the  four  DMUs  in  the  graphical  example,  there  are 
now  "n"  DMUs.  Each  of  these  DMUs  uses  ”m''  inputs  to  produce 
••s”  outputs.  Xij  is  the  eimount  of  input  •'i'*  used  by  DMUj. 

Yr j  is  the  amount  of  output  "r"  produced  by  DMUj .  The 
decision  variables  of  interest  are  the  unit  weights  attached 
to  each  of  the  inputs  and  outputs  used  by  DMUj.  V^j  is  the 
unit  weight  placed  on  input  ”i"  by  DMUj.  V^-j  is  the  unit 
weight  placed  on  output  "r”  by  DMUj  (49:17). 

With  the  constants  and  decision  variables  outlined,  the 
next  step  is  to  formulate  a  total  of  "n"  fractional  linear 
programs.  A  single  fractional  linear  program  is  formulated 
for  each  DMU.  The  specific  DMU  being  evaluated  is 
designated  as  DMUq  (39:121).  The  objective  function  of  the 
fractional  linear  program  is  the  ratio  of  the  total  weighted 
output  of  DMUq  divided  by  its  total  weighted  input  (13:430): 
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Each  DMU  must  choose  its  unit  weights  subject  to  the 
constraint  that  no  other  DMU  would  have  an  efficiency  rating 
greater  than  one  if  it  used  the  same  weights.  In  other 
words,  a  DMU  which  is  more  efficient  than  DMUq  will  receive 
DEA  rating  of  1.0  while  DMUq  will  be  forced  off  the 
efficiency  frontier  and  receive  a  DEA  rating  less  than  1.0. 
This  provides  a  fractional  constraint  such  that  (13:430): 
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Z  Uj^o  Yrj 
r=l 

^ 2  1;  j=l,2,...  n  (3) 

m 
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The  final  two  constraints  stipulate  that  the  selected 
unit  weights  can  not  be  negative.  These  constraints  are 
formulated  such  that  (13:430): 


Uj-o  s  0;  r=l,2, . . .  s  (4) 

Vj-Q  s  0 j  i*l ,2,...m  (5) 


The  Charnes.  Cooper.  Rhodes  (OCR)  Transformation 

In  Chapter  II,  Charnes,  Cooper,  and  Rhodes  were  given 
credit  for  developing  the  DEA  technique.  This  credit 
includes  a  transformation  method  to  permit  use  of  the 


59 


simplex  solution  algorithm.  The  eJsove  model  (equations  2-5) 
is  an  extended  nonlinear  programming  formulation  of  an 
ordinary  fractional  programming  problem  (13:431).  It 
requires  a  linear  transformation  in  order  to  solve  using 
ordinary  simplex  linear  programming. 

The  detailed  steps  of  the  CCR  transformation  are  not 
required  for  understanding  of  the  DEA  methodology.  The  CCR 
tremsformation  replaces  the  extended  nonlinear  fractional 
programming  problem  with  an  ordinary  linear  programming 
problem.  This  linear  program  is  the  DEA  methodology  used  in 
this  research.  The  model  is  shown  in  equations  6-10 
(13:432): 


s 

(DEA)  Mcixiraize  Hq  *  S  Uj-q  '^to  (6) 

r=l 

Subject  To: 

s  m 

^  Uro  Yj-j  -  Z  ^ij  ^  j=l,2,...n  (7) 

r=l  i=l 

m 

2  ^io  “  ^ 
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Uro  ^0;  r=l,2,...  s 
Vio  ^  Of  i=l/2,...  m 


Equation  (6)  is  the  objective  function  of  the  DEA 
methodology  employed  in  this  research.  Equations  (7) 
through  (10)  form  the  constraints.  Taken  together,  these 
equations  form  the  DEA  technique  which  can  be  solved  through 
the  ordinary  simplex  algorithm  (13:429).  In  simple  terms, 


(9) 

(10) 
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the  objective  function  (6)  attempts  to  m€aimize  the  weighted 
outputs  of  DMUq.  The  first  constraint  (7)  states  that  using 
DMUq’s  weights,  the  sum  of  the  weighted  outputs  minus  the 
sum  of  the  weighted  inputs  for  all  DMUs  in  the  model,  must 
be  less  than  or  equal  to  zero.  The  second  constraint  (8) 
stipulates  the  sum  of  the  weighted  inputs  for  DMUq  must 
equal  one.  The  final  two  constraints  (9  &  10)  ensure  the 
weights  are  assigned  some  value  greater  than  or  equal  to 
zero. 

Complete  PEA  Analysis 

A  complete  DEA  requires  the  solution  of  "n"  linear 
programs,  one  for  each  DMT),  using  the  transformed  function 
and  constraints.  These  linear  programs  are  similar  to  each 
other.  The  only  difference  is  in  the  objective  function 
coefficients  and  specification  of  the  last  constraint.  This 
characteristic  makes  the  computational  effort  required  to 
solve  the  DEA  problem  relatively  simple  after  initial  model 
specification  (49:18). 

Once  the  DEA  solution  is  obtained,  the  output  provides 
the  analyst  with  information  concerning  relative  efficiency, 
the  use  of  resources  to  produce  output,  and  methods  for 
performeuice  improvement.  The  nature  of  linear  programming 
provides  this  information  in  the  form  of  the  weights 
(coefficients)  assigned  to  the  decision  variables  and  the 
dual  (slack)  solution.  The  results  of  the  DEA  application 
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to  operational  contracting  squadrons  are  presented  in 
Chapter  IV. 


Pftgt  lit _ Specific  Ob-iectives  and  Investigative  Questionr 


A  complete  introduction  to  the  DEA  methodology  was 
required  before  addressing  the  specific  techniques  employed 
to  answer  the  research  objectives  euid  related  investigative 
questions.  The  DEA  technique  was  the  critical  factor 
driving  the  chronological  use  of  five  investigative 
questions  for  meeting  the  research  objectives.  Part  II  of 
this  chapter  addresses  the  specific  investigative  questions 
and  identifies  techniques  used  to  provide  solutions. 

Investigative  Questions  1  &  2;  What  are  the  Most 
Critical  Outputs  and  Resources?  The  first  step  in 
developing  the  DEA  model  was  to  identify  candidates  for 
possible  input  and  output  measures,  within  any  DEA 
application,  the  selection  of  inputs  and  outputs  is 
critical.  Clarke  suggests  examining  five  characteristics  of 
data  when  selecting  possible  input  and  output  measures 
(14:377).  These  characteristics  have  been  adapted  to  the 
operational  contracting  environment  and  are  listed  in  Table 


3.4. 
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TABLE  3.4  Charact«ristics  of  DBA  Varlablos 


1.  Data  should  be  regularly  available  to  (or  from) 
memagers  involved  in  operational  contracting  memagement. 

2.  Utilized  data  should  be  collected  emd  reported  in  a 
consistent  manner  for  all  contracting  squadrons. 

3.  Data  reported  in  ratio  form  should  be  separable 
into  numerator  and  denominator  values  so  differences  in 
scale  can  be  identified  when  necessary. 

4.  Data  should  be  numerical  or  easily  converted  into 
numerical  form. 

5.  All  numerical  values  should  be  managerially 
significemt/  meaningful,  emd  readily  understood  by 
operational  contracting  management. 

(14:377) 

The  data  provided  by  the  Base  Contracting  Automated 
System  (BCAS)  was  reviewed  in  detail.  All  possible  measures 
with  the  potential  to  meet  Clarke's  first  four 
characteristics  were  identified  and  listed.  A  complete  list 
of  possible  measures  is  included  as  Appendix  A.  The  initial 
list  revealed  155  possible  measures. 

The  next  step  in  identifying  input  and  output  measures 
was  to  determine  the  variables  which  were  meuiagerially 
significemt.  The  original  155  possible  measures  were  taken 
from  the  BCAS  report  entitled  RCS  HAF-RDC  (M&A)7106,  Base 
Contracting  Activity  Report  (referred  to  as  the  7106 
report ) .  These  measures  were  screened  by  a  group  of  eight 
former  operational  contracting  managers.  Each  had  managed 
within  the  last  twelve  months.  The  screening  process  was 
used  to  narrow  the  total  number  of  items  so  a  manageable  and 
administratable  survey  could  be  developed.  The  screening 
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process  resulted  in  the  selection  of  fifty  one  possible 
measures  which  could  be  meaningful,  managerially 
significant,  and  readily  understood  by  contracting 
management.  A  complete  list  of  the  fifty  one  possible 
measures  is  provided  in  Appendix  B. 

Once  identified,  these  measures  were  incorporated  into 
a  survey  using  a  five  point  Likert  scale.  The  scale 
consisted  of  the  following  five  points:  (1)  Definitely  Not 
Important,  (2)  Somewhat  Not  Important,  (3)  Neutral,  (4) 
Somewhat  Important,  (5)  Definitely  Important.  Management 
literature  often  assiunes  a  Likert  scale,  with  a  true  neutral 
response,  provides  data  with  order  and  distance.  In  other 
words,  the  Likert  scale  is  assumed  to  represent  an  interval 
scale  (21:222)  Once  this  assumption  is  made,  the  research 
can  employ  parametric  statistics  to  measure  central  tendency 
and  conduct  hypothesis  tests.  There  exists  wide 
disagreement  concerning  this  treatment  of  Likert  responses 
as  interval  data  (22:175). 

The  current  research  assumes  interval  data  resulting 
from  the  survey  to  identify  critical  input  and  output 
variables.  Parametric  statistical  tests  were  performed  to 
identify  those  variables  important  to  contracting  managers. 
This  assumption  does  not  remove  the  possibility  of 
inequality  of  intervals  on  the  Likert  scale  (22:175). 
However,  the  only  purpose  of  the  survey  was  to  reduce  the  51 
possible  variables  to  those  of  relative  importance.  Once 
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reduced,  qualitative  emalysis  was  conducted  in  order  to 
construct  the  final  DEA  model. 

During  survey  construction,  it  was  discovered  that 
emother  thesis  effort  would  be  issuing  a  similar  instmment 
to  the  same  sample.  In  order  to  improve  the  response  rate 
for  both  tnesis  projects  ^uld  minimize  effort  on  the  part  of 
respondents,  the  two  surveys  were  combined  into  a  single 
survey.  A  copy  of  the  survey  is  included  in  Appendix  C. 

All  questions  used  for  this  thesis  effort  are  typed  in  bold 
print . 

The  survey  was  pilot  tested  on  a  sample  of  ten  former 
operational  contracting  managers.  Their  duties  ranged  from 
serving  as  the  Squadron  Commander  to  Chiefs  of  Contracting 
Flights.  In  addition  to  the  pilot  test,  the  survey  was 
reviewed  by  personnel  at  the  Air  Force  Logistics  Management 
Agency  at  Maxwell  AFB,  AL  (AFMLA/LGC).  This  organization 
was  chosen  because  its  mission  is  conducting  management 
research  concerning  operational  contracting  performance. 

Once  pilot  and  review  comments  were  incorporated,  the 
survey  was  administered  to  the  selected  sample.  Because  a 
primary  goal  was  to  identify  input  and  output  measures  which 
were  managerially  significant,  the  population  of  interest 
was  all  members  of  operational  contracting  management  in  the 
Air  Force.  In  order  to  properly  generalize  the  survey 
findings  to  this  population,  a  modified  method  of  quota 
sampling  was  used  (21:275). 
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Two  control  dimensions  were  used  for  the  quota  sampling 
technique.  The  first  dimension  was  a  sample  of  all 
operational  contracting  squadrons  within  the  Continental 
United  States  (CONUS).  The  decision  to  sample  all  82 
operational  contracting  squadrons  eliminated  problems 
associated  with  selecting  random  squadrons  from  the 
population.  The  second  dimension  was  to  specify  meinagement 
as  specific  job  levels  within  each  operational  contracting 
sc[uadron.  The  second  dimension  limited  the  sample  to  the 
top  three  management  positions  in  each  squadron.  These 
positions  include  Squadron  Commanders  (also  known  as  Base 
Contracting  Officers  (BCOs)),  Deputy  BCOS/  and  Squadron 
Executive  Officers/NCOs.  These  positions  were  selected 
because  they  represent  operational  contracting  managers  with 
the  responsibility  of  resource  control  and  overall 
performance  at  each  squadron. 

Once  management  was  identified,  the  survey  was 
administered  to  a  sample  consisting  of  all  operational 
contracting  managers  within  the  CONUS.  The  sample  included 
mcuiagement  at  all  squadrons  in  Air  Combat  Command  (ACC),  Air 
Force  Material  Command  (AFMC),  Air  Mobility  Command  (AMC), 
Air  Force  Space  Command  (AF  Spacecom),  and  Air  Training 
Comm^md  (ATC).  The  management  sample  included  two  hundred 
fifty  eight  possible  respondents.  In  effect,  this  process 
reflected  a  census  of  the  top  three  operational  contracting 
managers . 
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After  the  management  suin^eys  were  returned,  two 
statistical  techniques  were  used  to  identify  critical  input 
emd  output  measures  to  use  in  the  DEA  application.  First, 
the  questions  covering  the  51  possible  measures  were  rank 
ordered  based  upon  the  mecm  response.  The  variemce  and 
standard  deviations  were  calculated  for  each  question. 

These  measures  gave  a  general  indication  of  the  amount  of 
dispersion  in  the  responses  to  a  given  question.  However,  a 
more  stringent  technique  was  required  to  identify  those 
vari£d>les  meeting  Clarke's  fifth  characteristic  of 
managerial  significance  ( 14 : 377 ) . 

Because  the  survey  included  258  possible  respondents,  a 
Large-'Sample  Test  of  Hypothesis  About  a  Population  Mean  was 
chosen  as  the  method  to  identify  which  of  the  51  possible 
variables  to  include  in  the  DEA  model.  The  mean  response 
for  each  question  was  compared  to  the  Likert  scale  response 
of  4.0.  A  one-tailed  test  (a=.05)  was  employed  in  order  to 
identify  those  variables  whose  mean  response  was 
significantly  greater  than  the  survey  response  indicating  a 
somewhat  important  variable  (37:356).  An 'example  of  the 
Large  Sample  Test  of  Hypothesis  is  shown  in  Table  3.5. 
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TABLE  3.5  Large  Saaple  Test  of  Hypothesis  About  m 

One-Tailed  Test  a  =  .05 

Hq:  ^  individual  question  »  4 
|i  individual  question  >  4 

Test  Statistic:  z  =  X  -  4 

a_ 

X 

Where:  a_  *  o/Vn 
X 

Rejection  Region:  z  <  -z^ 

(or  z  >  z„  when  Ha:  n  > 

(37:356) 

There  are  no  assumptions  required  cdiout  the  underlying 
probe±>ility  distribution  when  using  the  Large  Sample  Test  of 
Hypothesis.  For  large  samples  (n  >  30),  the  Central  Limit 
Theorem  provides  assurance  that  the  test  statistic  will  be 
approximately  normally  distributed.  This  theorem  holds 
regardless  of  the  shape  of  the  underlying  probability 
distribution  (37:356).  Additionally,  the  decision  to  test 
mean  responses  away  from  the  Likert  response  of  4.0,  insured 
the  identification  of  truly  important  vari€d3les. 

Once  the  original  51  possible  variables  were  reduced 
using  the  Large  Sample  Test  of  Hypothesis,  the  managerially 
significant  variables  were  further  reduced  through  non- 
quantitative  examination  based  upon  Clarke's  first  four 
characteristics  listed  in  Table  3.4.  This  reduction,  using 
Clarke's  characteristics,  ensured  only  important  variables. 
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which  managers  could  influence,  were  included  in  the  DEA 
model . 

Investigative  Question  3:  What  are  the  Results  of  the 
DEA  Application  to  Operational  Contracting  Squadrons?  After 
identifying  the  input  and  output  measures,  the  third 
research  (question  was  addressed.  This  question  required  the 
actual  formulation  of  a  DEA  model  based  on  the  identified 
inputs  and  outputs.  This  methodology  required  execution  of 
a  linear  program  for  each  of  the  DMUs  under  analysis.  The 
results  of  the  DEA  application  are  presented  in  tcdjular 
format.  These  results  include  the  DEA  or  efficiency  score 
for  each  contracting  squadron  and  the  ERS  for  each  squadron 
receiving  a  score  less  than  one.  The  concept  and 
formulation  of  DEA  was  discussed  in  Part  I  of  this  chapter, 
however,  a  description  of  the  archival  data  source  is 
required. 

All  potential  input  and  output  measures  were  derived 
from  data  available  on  the  BCAS  7106  report  of  each 
contracting  squadron.  Once  survey  results  identified 
measures  of  greatest  importance,  the  data  was  collected  from 
7106  reports  from  all  ACC  and  AFMC  bases  for  fiscal  year 
1992.  This  allowed  application  of  the  DEA  model  to  forty 
five  operational  contracting  squadrons  ensuring  a  large 
sample  of  DMUs  for  analysis. 
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Investigative  Question  4;  What  Information  Does  PEA 
Provide  the  Air  Force  Operational  Contracting  Manager?  The 


euiswer  to  investigative  question  four  was  provided  by 
interpreting  the  results  of  DEA  for  each  operational 
contracting  squadron.  DEA  provides  the  manager  with 
information  to  aid  in  performance  improvement.  The  purpose 
of  investigative  question  four  was  to  demonstrate  the 
information  DEA  provides  contracting  managers  in  addition  to 
the  basic  efficiency  score  and  ERS.  This  information 
includes  the  four  main  areas  listed  in  Table  3.6. 

TABLE  3.6  Management  Information  Provided  by  DEA 


1.  The  input  and  output  weights  assigned  by  DEA  for 
each  operational  contracting  squadron. 

2-  The  comparison  of  relatively  inefficient  squadrons 
with  its  ERS  by  calculating  the  excess  inputs  used  to 
produce  the  given  level  of  output. 

3.  The  calculation  of  a  Hypothetical  Comparison  Unit 
(HCU)  for  each  squadron  receiving  a  DEA  rating  less  than 

1.0. 

4.  The  method  to  examine  improvement  through  reduction 

of  a  single  input. _ _ 

Each  of  these  four  areas  of  information  provide 
managers  with  methods  of  improving  performance.  A  complete 
discussion  of  how  DEA  generates  the  four  areas  of  management 
information  is  provided  in  Chapter  IV.  In  order  to 
demonstrate  how  DEA  generates  this  information,  Wurt smith 
AFB  was  selected  as  the  contracting  squadron  to  examine  in 
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Chapter  IV.  The  information  for  the  remaining  relatively 
inefficient  squadrons  is  presented  in  Appendix  D. 

Investigative  Question  5:  What  Characteristics  of  a 
Performance  Measurement  System  are  Desired  bv  Operational 
Contracting  Managers  and  How  are  They  Addressed  by  PEA?  The 
answer  to  this  question  was  critical  to  determining  DEA's 
potential  use  by  operational  contracting  managers.  The 
methodology  employed  to  analyze  this  question  was  a  two-step 
process.  First,  desircdjle  characteristics  of  a  performance 
measurement  system  were  identified  through  the  survey  of 
operational  contracting  managers.  The  same  survey  used  to 
analyze  input  and  output  varied^les  contained  eleven 
questions  concerning  desired  characteristics.  These 
questions  allowed  contracting  managers  to  respond  on  the 
same  five  point  Likert  scale.  The  survey  instrument  is 
provided  in  Appendix  C.  A  list  of  the  eleven  questions 
analyzed  to  determine  management's  desired  characteristics 
of  a  performance  measurement  system  are  shown  in  Table  3.7. 
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TABLE  3.7  Survey  Questions  Responding  to  IQ  5 


#  Question 

Strongly  Agree  Disagree  Neutral  Agree  Strongly  Agree 

1  2  3  4  5 

8  The  current  Command  IG  evaluation  system  satisfactorily  measures 
organizational  performance. 

9  An  evaluation  system,  different  from  the  Command  IG,  would  be  useful  in 
measuring  overall  organizational  performance. 

#  Question 

Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  In^ortant  Neutral  In^ortant  Important 

1  2  3  4  5 

98  The  current  IG  method  of  performance  evaluation  provides  feedback  to 
help  improve  performance  which  is? 

99  Current  Command  euid  Air  Force  awards  provide  feedback  concerning 
performance  which  is? 

100  Improving  operational  contracting  squadron  productivity  and  efficiency 
is  a  goal  which  is? 

101  A  contracting  evaluation  system  which  simultaneously  evaluates  several 
inputs  and  outputs  to  the  process  (as  opposed  to  single  measures  such  as 
CALT) ,  is  a  tool  which  is? 

102  A  cbntracting  evaluation  system  which  con^ares  all  operational 
contracting  squadrons  while  taking  into  account  differences  in  squadron 
characteristics  (such  as  manning,  experience,  and  workload) ,  is  a  tool 
which  is? 

103  A  contracting  evaluation  system  which  does  not  rely  solely  upon 
measures  which  the  chain  of  command  (LG,  Wing  CC)  find  important 
is  a  tool  which  is? 

104  A  contracting  evaluation  system  which  compares  squadrons  relative  to 
other  contracting  squadrons,  is  a  tool  which  is? 

105  A  contracting  evaluation  system  which  con^ares  contracting  squadrons  to 
the  best  performers  as  opposed  to  the  average,  is  a  tool  which  is? 

106  A  contracting  evaluation  system  which  provides  managers  with  timely 
feedback,  including  exact  data  on  resourc-  utilization  and  relative 

_ efficiencies  compared  with  other  squadrons,  is  a  tool  which  is? 


Once  survey  results  were  obtained,  the  data  were 
analyzed  using  a  methodology  similar  to  that  listed  in  Table 
3.5.  A  Large  Sample  Test  of  Hypothesis  About  a  Population 
Mean  was  used  to  determine  the  significance  of  the  mean 
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response  for  each  of  the  eleven  questions.  Because  the 
objective  of  this  test  was  to  examine  both  positive  and 
negative  responses,  a  two-tailed  test  was  employed.  The 
research  was  concerned  with  any  significant  differences 
ed)ove  or  below  the  neutral  Likert  scale  response  of  3.0.  An 
exeunple  of  this  test  is  provided  in  Table  3.8. 

TABLE  3.8  Large  Sample  Test  of  Hypothesis  About  ^ 

Two-Tailed  Test  a  =  .05,  a/2  =  .025 
^  individual  question  =  3 
^  individual  question  *  3 

Test  Statistic;  z  =  X  -  3 
•  a 

X 

Where;  <t_  =  cr/Vn 
X 

Rejection  Region;  z  <  -Za/2 

or  z  >  z^/2 

(37;357) 

The  decision  to  test  mean  responses  away  from  the 
neutral  response  was  made  to  ensure  identification  of  any 
performance  evaluation  characteristic  desired  by  managers. 
The  purpose  of  the  eleven  questions  was  to  discover  opinions 
cuid  feelings  edsout  current  and  desired  methods  to  measure 
operational  contracting  performance.  The  goal  was  to  ensure 
that  relatively  significant  management  desires  were 
discussed  in  the  results  of  the  research. 
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Performance  evaluation  characteristics  identified  as 
important  by  managers  were  compared  to  the  DEA  results. 

This  comparison  was  made  by  addressing  the  portions  of  DEA 
output  which  may  aid  operational  contracting  management  to 
improve  overall  efficiency  and  performance.  If  the  DEA 
model  provided  feedback  and  addressed  the  characteristics 
identified,  it  could  be  a  useful  addition  to  or  replacement 
for  current  performance  evaluation  systems. 

Chapter  Summary 

This  chapter  has  provided  an  explanation  of  the 
research  design  auid  related  DEA  methodology.  The  chapter 
was  divided  in  two  parts.  The  first  part  introduced  the 
primary  DEA  methodology.  A  complete  list  of  DEA  assumptions 
was  initially  provided  to  establish  later  model  development. 
A  simple  graphical  example  using  the  same  four  hypothetical 
contracting  squadrons,  introduced  in  Chapter  II,  allowed 
development  of  the  efficient  frontier  concept.  The  complete 
DEA  linear  programming  model  was  then  developed.  This  model 
allows  for  analysis  of  multiple  inputs  and  multiple  outputs 
for  operational  contracting  squadrons. 

Part  II  of  the  chapter  provided  the  various  methods 
employed  to  meet  each  research  objective.  Each  of  the  five 
investigative  questions,  with  specific  analysis  techniques, 
were  addressed.  The  overall  objective  of  developing  an 
alternate  method  of  measuring  operational  contracting 
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performemce  will  be  achieved  by  answering  each  specific 
investigative  question  in  Chapter  IV. 
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12.  RgSWlts 


Overview 

The  purpose  of  this  chapter  is  to  provide  the  results 
of  the  methodology  employed  to  cmswer  each  of  the 
investigative  questions.  The  results  are  presented  in  a 
two-step  fashion.  Part  I  of  this  chapter  provides  the 
results  to  investigative  questions  one  amd  two.  Part  I 
describes  the  selection  of  the  most  critical  outputs  and 
resources  of  an  operational  contracting  squadron.  The 
selection  of  resources  and  outputs  was  necessary  to 
construct  an  appropriate  DEA  model  and  answer  the  remaining 
investigative  questions . 

Part  II  of  this  chapter  answers  investigative  questions 
three  through  five.  The  major  portion  of  these  answers  were 
discovered  through  an  application  of  DEA  to  the  DMUs 
selected  from  ACC  and  AFMC.  After  answering  each  of  the 
investigative  questions  in  Part  I  and  Part  II  of  this 
chapter,  the  primary  objectives  of  this  research  were  met. 
These  objectives  included  developing  a  DEA  model  for 
operational  contracting  squadrons  and  evaluating  DEA's 
output  cuid  potential  use  to  managers  for  performance 
evaluation. 

I: _ Selection  of  Critical  Resources  and  Outputs 

Before  a  DEA  model  for  operational  contracting 
squadrons  could  be  constructed,  critical  resources  and 
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outputs  to  the  contracting  process  were  identified.  These 
variables  were  later  incorporated  into  the  DEA  model.  This 
section  of  the  chapter  describes  the  management  survey 
results  and  the  selection  of  critical  resources  emd  outputs. 
This  section  provides  answers  to  the  first  two  investigative 
questions  which  are  listed  in  Table  4.1. 

TABLE  4.1  Investigative  Questions  142 

IQ  1:  What  are  the  most  critical  outputs  generated 
from  the  operational  contracting  fimction? 

IQ  2:  What  are  the  most  critical  resources  available 
_ to  operational  contracting  squadrons? _ 


General  Survey  Results.  The  management  survey,  used  to 
identify  the  variables  to  include  in  the  DEA  model,  was  sent 
to  258  possible  respondents.  Of  these  258  surveys,  165  were 
returned.  The  overall  return  rate  was  64%.  Of  the  165 
returned  surveys,  only  153  were  actually  used  in  this  study. 
Twelve  surveys  were  determined  unusable  in  this  research. 
Each  of  the  twelve  unusable  suzrveys  were  incomplete  due  to 
the  respondent's  lack  of  experience  in  operational 
contracting.  The  total  usable  return  rate  was  59%. 

The  first  seven  questions  on  the  survey  provided 
general  demographic  information.  This  information  was  not 
used  in  the  actual  DEA  formulation,  but  it  does  provide 
information  concerning  the  background  of  the  managers 
responding.  Table  4.2  provides  demographic  information 
gained  from  the  survey. 


77 


TABLE  4.2  General  Deaographics 


Average  Contracting  Experience 

15.12 

Average  Experience  in  Current  Position 

2.98 

Number  of  Officers  Responding 

51 

Number  of  Civilians  Responding 

52 

Number  of  Enlisted  Responding 

50 

Percent  in  Same  Job  During  Last  IG 

48.40 

Number  of  Respondents  from  AMC 

25 

Number  of  Respondents  from  ACC 

77 

Number  of  Respondents  from  ATC 

23 

Number  of  Respondents  from  AFMC 

13 

Number  of  Respondents  from  Other  Commands 

15 

Identification  of  Critical  Resources  and  Outputs.  The 
main  purpose  of  the  management  survey  was  to  identify  those 
resources  and  outputs  m2uiagement  deemed  critical  to  the 
operational  contracting  process.  These  critical  variables 
could  then  be  included  in  a  DEA  model  to  measure  a 
contracting  squadron's  productivity.  The  survey  allowed 
contracting  managers  to  rank  each  of  the  51  possible 
variables/  from  definitely  not  important  to  definitely 
importemt,  on  a  five  point  Likert  scale.  The  numerical 
results  were  then  used  to  identify  those  varied>les  which 
.were  meuiagerially  significant. 

In  order  to  analyze  the  resources  and  outputs  the  mean 
response  to  each  question  was  remk  ordered.  Results  of  this 
process  are  shown  in  Table  4.3. 
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TABLE  4.3  Naan  Response  to  Possible  Resources  and  Outputs 


Oiiestioo# 

Descriptioo 

Mean  Response 

56 

Total  Squadron  Experience  Level 

4.59 

25 

Personnel  Assigned  vs.  Authorized 

4.52 

15 

Number  of  Contracts  Behind  Schedule 

4.50 

62 

Total  Number  of  Line  Items  Received 

4.37 

90 

Total  Number  of  Active  Service  Contracts 

4.37 

88 

Total  Number  of  Active  Contracts 

4.36 

50 

Number  of  Buyers/Contract  Administrators 

4.31 

61 

Total  Number  of  Line  Items  Received  Priority  1-8 

4.31 

89 

Total  Active  Construction  and  Arcb**ect  &  Engineering  Contracts 

4.27 

91 

Total  Number  of  Active  Commodities  Contracts 

4.25 

58 

Total  Experience  Level  Without  Admiitistrative  or  Clerical  Support 

4,20 

51 

Number  of  Clerical  Personnel 

4.18 

26 

Total  Number  of  Contracting  Actions 

4.16 

52 

Number  of  Management  Personnel 

4.16 

24 

Percent  Competitive  Action  Measure 

4.14 

53 

Number  of  Assigned  Civilians 

4.14 

65 

Total  Number  of  Modifications  Executed 

4.14 

57 

Average  Office  Experience  Level 

4.12 

59 

Average  Office  Experience  Level  Without  Clerk  or  Administrative 

Support 

4.11 

55 

Number  of  Assigned  Enlisted  Personnel 

4.10 

29 

Total  Dollars  Awarded  Competitively 

4.10 

63 

Number  of  Different  Customer  Organizations  Served 

4.04 

66 

Priority  1  -  3  CALT 

4.01 

74 

Total  Number  of  Line  Items  Awarded 

4.01 

27 

Total  Number  of  Centralized  Actions 

4.01 

75 

Total  Number  of  Centralized  Line  Items  Awarded 

3.98 

87 

Percent  Competitive  Dollars  Awarded  Measure 

3.95 

67 

Priority  4-8  CALT 

3.93 

84 

Total  Number  of  Large  Business  Actions  Available  for  Small  Business 

3.93 

85 

Total  Large  Business  Dollars  Awarded  That  Were  Available  for  Small 
Business 

3.87 

69 

Total  Number  of  Decentralized  Actions 

3.87 

68 

Total  Centralized  Dollars  Awarded 

3.83 

54 

T(gal  Dollars  Awarded 

3.81 

73 

Total  Section  8A  (Small  Disadvantaged)  Dollars  Awarded 

3.78 

83 

Total  Set-Aside  Actions 

3.77 

86 

Small  Business  Dollars  Awarded  Divided  1^  the  Total  Dollars  Available 
for  Small  Business 

3.77 

28 

Total  Number  of  Decentralized  Actions 

3.77 

77 

Total  Centralized  Dollars  Awarded 

3.76 

64 

Total  Dollars  Awarded 

3.76 

72 

Total  Number  of  Section  8  A  (Small  Disadvantaged)  Actions 

3.76 

82 

Total  Modifications  Divided  by  Centralized  Actions  Awarded 

3.73 

33 

Number  of  Blanket  Purchase  Agreements  Administered 

3.64 

79 


TABLE  4 . 3  Continued 


OuCStHMI# 

Description 

Mean  Response 

76 

Total  Number  of  Decentralized  Line  Items  Awarded 

3.61 

78 

Total  Decentralized  Dollars  Awarded 

3.48 

92 

Priced  Actions  Divided  by  Unpriced  Actions 

3.42 

79 

Centralized  Actions  Divided  by  Centralized  Line  Items 

3.41 

12 

Last  Inspector  General  Rating 

3.35 

81 

Decentralized  Line  Items  Awarded  Divided  by  Centralized  Line  Items 
Awarded 

3.24 

80 

Decentralized  Dollars  Awarded  Divided  by  Total  Dollars  Awarded 

3.24 

70 

Total  Number  of  Non'Apprm>riated  Actions  Awarded 

2.96 

71 

Total  Non-Appropriated  Dollars  Awarded 

2.94 

R2mk  ordering  the  possible  resources  and  outputs  by 
mean  response  provided  a  general  indication  of  each 
variables  importance  to  the  operational  contracting  manager. 
However,  a  stronger  statistical  approach  was  needed  in  order 
to  reduce  the  possible  variables  to  those  managers  indicated 
as  significant.  A  Large-Sample  Test  of  Hypothesis  About  a 
Population  Mean  was  conducted  using  the  z  statistic 
(37:356) . 

The  mean  response  for  all  51  questions  was  compared  to 
the  Likert  scale  response  of  4.0.  A  response  of  4.0 
indicated  a  variable  that  was  somewhat  important  to  the 
contracting  manager.  A  one-tailed  test  was  employed  in 
order  to  identify  those  variables  whose  mean  response  was 
significantly  greater  than  the  somewhat  important  survey 
response.  The  purpose  of  this  comparison  was  to  further 
reduce  the  number  of  critical  resources  and  outputs  based 
upon  the  management  survey. 
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In  order  to  provide  an  example  of  the  application  of  a 
large  seunple  test  of  hypothesis,  the  calculations  for  survey 
question  56  are  provided  in  Ted)le  4.4.  This  survey  question 
covered  the  importance  of  total  office  experience  level  to 
the  operational  contracting  meuiger.  As  the  rank  ordering  of 
meem  responses  demonstrated,  this  question  was  ranked  most 
important  among  those  managers  returning  surveys. 

TABLE  4.4  Hypothesis  Test  for  Question  56 

One-Tailed  Test  a  =  .05 

Ho:  ^  question  56  =  4 

Ha:  ^  question  56  >  4 

Test  Statistic:  z  =  X  -  4 

o_ 

X 

Test  Statistic:  z  =  4.595  -  4 

.057495 

Z  =  10.35 

Rejection  Region:  z  >  z^ 

10.35  >  1.645 

Conclusion:  Reject  Null  Hypothesis _ 


The  process  of  conducting  large  sample  tests  of 
hypothesis  about  a  population  mean  was  continued  for  all  51 
possible  resources  and  outputs.  An  a  value  of  .05  was 
chosen  to  identify  only  those  variables  managerially 
important  at  the  95%  significance  level.  This  process 
reduced  the  niunber  of  variables  to  16  possible  resources  and 
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outputs  to  be  included  in  the  OEA  model.  A  list  of  the 
variables  considered  for  use  in  DEA,  along  with  their  test 
statistics  and  associated  significance  level  is  provided  in 
Tedjle  4.5. 


TABLE  4.5  Variables  Considered  for  DBA  Use 


Question  # 

Description 

TestStat 

56 

Total  Squadron  Experience  Level 

10.35 

• 

25 

Personnel  Assigned  vs.  Authorized 

8.20 

• 

15 

Number  of  Contracts  Behind  Schedule 

8.17 

* 

62 

Total  Number  of  Line  Items  Received 

5.81 

* 

88 

Total  Number  of  Active  Contracts 

5.03 

* 

90 

Total  Number  of  Active  Service  Contracts 

5.02 

• 

61 

Total  Number  of  Priority  1-8  Line  Items  Received 

4.50 

• 

50 

Number  of  Buyers/Contract  Administrators 

4.35 

• 

89 

Total  Number  of  Active  Construction  and  Architect  & 
Engineering  Contracts 

3.64 

• 

91 

Total  Number  of  Active  Commodities  Contracts 

3.34 

• 

51 

Number  of  Clerical  Personnel 

2.54 

* 

26 

Total  Number  of  Contracting  Actions 

2.48 

* 

52 

Number  of  Management  Personnel 

2.15 

• 

24 

Percent  Competitive  Action  Measure 

2.14 

• 

65 

Total  Number  of  Modifications  Executed 

2.07 

• 

29 

Total  Dollars  Awarded  Competitively 

1.72 

* 

•  Significant  at  the  95%  Level 

Testing  the  difference  between  means  revealed  16 
possible  variables  significant  at  the  95%  level.  All  16 
variables  were  considered  for  use  in  the  DEA  model  because 
operational  contracting  managers  had  identified  them  as 
significant  resources  and  outputs  to  the  operational 
contracting  process.  These  varicdiles  also  met  several  other 
characteristics  outlined  by  Clarke  and  listed  in  Table  3.4. 
For  example,  the  16  possible  variables  are  available  to 
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contracting  managers.  The  variables,  reported  in  ratio 
form,  are  separ2d3le  into  numerator  and  denominator  values. 
All  variables  are  numerical  in  nature.  Finally,  the  16 
possible  variables  are  readily  understood  by  contracting 
mauiagement  because  they  were  chosen  from  the  7106  report. 

Critical  Variables  Chosen  for  PEA  Model.  The  16 
resources  and  outputs  identified  as  significant  by 
operational  contracting  managers  also  met  the  majority  of 
Clarke's  characteristics  for  inclusion  into  a  DBA  model. 

The  final  variable  selection  was  made  using  two  important 
criteria  listed  by  Clarke.  First,  all  data  must  be 
collected  and  reported  in  a  consistent  manner  for  all 
operational  contracting  squadrons.  Second,  all  variables 
chosen  must  be  managerially  meaningful  (14; 377).  This 
second  criteria  demanded  that  variables  chosen  must  not  only 
be  rated  as  significant,  but  also  have  a  strong  impact  on 
productivity  and  be  controllcible  by  management. 

These  final  two  criteria,  along  with  the  elimination  of 
variables  providing  redundant  information,  allowed  the  16 
possible  variables  to  be  reduced  to  9  outputs  and  resources 
to  the  contracting  process.  The  following  two  sections 
describe  each  of  the  9  variables  chosen  for  the  DBA  model. 
These  sections  provide  rationale  for  the  use  of  these  9 
variables. 

10  1;  Critical  Outputs  of  the  Contracting 
Process.  Of  the  9  varieibles  included  in  the  DBA  model,  five 
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were  outputs  to  the  contracting  process.  The  first  of  the 
five  outputs  was  the  number  of  active  service  contracts 
administered  (SERVICE)  by  each  contracting  squadron.  This 
output  was  measured  using  each  squadron's  7106  report.  The 
SERVICE  variable  measured  the  niunber  of  active  service 
contracts  on  hand  as  of  30  September  1992.  The  contracting 
manager  responses  to  the  survey  indicated  this  variable  was 
significant  at  the  95%  level. 

The  second  output  variable  chosen  was  the  niunber  of 
active  construction  and  architect  and  engineering  contracts 
administered  (CONST)  by  each  contracting  squadron.  This 
data  was  also  available  on  the  7106  report.  The  CONST 
variable  measured  the  niunber  of  active  construction  and 
architect  and  engineering  contracts  on  hand  as  of  30 
September  1992.  The  contracting  manager  responses  to  the 
survey  indicated  this  variable  was  significant  at  the  95% 
level. 

The  third  output  variable  chosen  was  the  number  of 
active  commodities  contracts  administered  (COMOD)  by  each 
contracting  squadron.  Data  for  each  squadron's  active 
commodities  contracts  was  availcd^le  on  their  respective  7106 
report.  Contracting  managers  indicated  this  was  significant 
at  the  95%  level.  It  is  important  to  note  that  the  COMOD 
variable,  like  SERVICE  and  CONST,  is  measured  at  a  specific 
point  and  time.  The  current  7106  report  does  not  provide  a 
cumulative  weighted  average  of  active  contracts  throughout  a 
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fiscal  year.  All  three  variables  were  measured  as  of  30 
September  1992. 

The  fourth  output  variable  chosen  was  the  total  number 
of  contracting  actions  executed  (ACTION)  by  each  squadron. 
This  data  was  availcd>le  on  each  squadron's  7106  report.  The 
total  actions  figure  used  was  the  cumulative  number  of 
actions  throughout  fiscal  year  1992.  This  includes 
centralized  and  decentralized  actions  performed  by  each 
contracting  squadron.  Decentralized  actions  are  those 
contracting  actions  conducted  by  other  members  of  the  base 
population.  For  example,  a  hospital  representative  may  have 
the  author!  1;y  (delegated  by  the  contracting  squadron)  to 
make  small  dollar  supply  purchases  without  going  '  hrough  the 
contracting  squadron.  The  contracting  squadron  retains 
administrative  control  and  reporting  responsibility  for 
these  decentralized  actions.  Once  again,  the  ACTION 
variable  was  significant  at  the  95%  level. 

The  fifth  and  final  output  used  in  the  DEA  model  was 
the  total  dollars  each  squadron  obligated  using  competitive 
procedures  ( COMP ) .  Contracting  managers  ranked  this  and  a 
variation  of  this  variable,  the  percent  competitive  dollars 
measure,  significant  at  the  95%  level.  The  percent 
competitive  dollars  measure  is  difficult  to  separate  into  a 
numerator  and  denominator  figure.  Additionally,  the  chosen 
total  competitive  dollars  obligated  variable  is  more  robust 
as  it  gives  a  better  indication  of  the  magnitude  of 
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competitive  purchasing  activity  at  each  squadron.  Finally, 
because  it  is  a  raw  measure,  not  requiring  separation  into 
numerator  and  denominator,  it  is  easier  to  interpret  and 
more  meaningful  for  contracting  managers.  Both  variations 
of  this  same  measure  were  considered  significant  by 
contracting  managers.  The  COMP  variable  chosen  for  DEA 
provides  more  meaningful  information  than  the  percent 
competitive  dollars  measure  omitted. 

IQ  2  i _ Critical  Resources  of  Operational 

Contracting.  After  selecting  five  critical  outputs,  the  DEA 
methodology  mandated  the  selection  of  various  critical 
inputs  to  the  contracting  process.  A  total  of  four  inputs 
were  chosen  for  this  application  of  DEA.  The  first  of  these 
inputs  was  the  total  office  experience  level  (EXP). 
Contracting  managers  selected  this  as  the  most  critical 
resource  to  the  operational  contracting  process.  The  EXP 
variable  was  determined  significant  at  the  95%  level.  Each 
bases  7106  report  contains  a  manpower  data  section.  Each 
individual  assigned  to  the  squadron  is  listed  in  this 
section  along  with  each  individual's  contracting  experience. 

The  second  input  chosen  for  the  DEA  model  was  the 
number  of  buyers/administrators  (BUYER)  assigned  to  each 
squadron.  This  varieUDle  was  determined  significant  at  the 
95%  level.  The  manpower  section  of  each  7106  report  assigns 
a  position  title  and  Air  Force  Specialty  Code  (AFSC)  to  each 
individual  in  the  squadron  which  denotes  contracting 
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responsibility.  This  information  was  used  to  identify  those 
members  of  the  squadron  who  were  neither  part  of  management 
or  clerical  and  administrative  support.  The  total  number  of 
buyers  and/or  contract  administrators  were  then  counted  for 
each  contracting  squadron. 

The  third  input  chosen  for  the  DEA  model  was  the  number 
of  clerical  or  administrative  support  (CLERK)  personnel 
assigned  to  each  squadron.  This  was  determined  significant 
at  the  95%  level.  The  CLERK  vari2d}le  includes  procurement 
clerks,  secretaries,  aind  squadron  administrative  support 
personnel.  Once  again,  the  data  for  individual  squadrons 
was  calculated  from  the  meuipower  section  of  the  7106  report. 

The  fourth  and  final  input  chosen  for  the  DEA  model  was 
the  number  of  management  personnel  assigned  (M6T)  to  each 
squadron.  Survey  results  indicated  this  variable  was 
significant  at  the  95%  level.  Data  for  this  variable  was 
also  obtained  from  the  manpower  section  of  the  7106  report. 

The  process  of  reducing  the  number  of  input  and  output 
measures  concentrated  on  providing  contracting  managers  with 
meaningful  feedback  concerning  squadron  productivity  and 
efficiency.  Each  of  the  16  possible  variables  were 
managerially  significant  at  the  95%  level.  The  simple 
approach  of  choosing  variables  with  the  highest  mean  survey 
results  would  not  have  satisfied  the  objective  of  giving 
operational  contracting  managers  meaningful  feedback.  For 
ezeunple,  the  input  varieibles  of  BUYER,  CLERK,  and  MGT  were 
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chosen  because  managers  do  have  some  control  over  the  mix  of 
personnel  in  the  squadron.  Because  of  the  managerially 
meaningful  objective,  some  variables  with  higher  mean  survey 
responses,  and  corresponding  test  statistics,  were  not 
chosen  for  inclusion  in  the  DEA  model. 

Process  Model  for  PEA.  Before  discussing  the  12 
significant  variables  not  used  in  the  DEA  model,  it  is 
important  to  summarize  the  model  chosen.  A  total  of  9 
v£u:iables  were  chosen  using  Clarke's  characteristics  listed 
in  Table  3.4.  Five  of  these  variadsles  were  outputs  and  four 
were  inputs  to  the  contracting  process.  The  operational 
contracting  process  modeled  in  this  DEA  application  is  shown 
in  Figure  4.1. 
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FIGURE  4.1  Operational  Contracting  DBA  Model 


Significant  Variables  Omitted  from  the_DEA  Model. 

After  identifying  those  variables  chosen  for  use  in  the  DEA 
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model,  it  is  important  to  outline  the  rationale  for  not 
including  the  other  seven  variables.  There  were  primarily 
three  reasons  for  excluding  certain  variables.  First,  many 
of  the  variables  provided  information  redundant  with  respect 
to  measures  chosen.  Second,  some  variables  did  not  justify 
inclusion  into  the  DEA  model  because  they  failed  to  provide 
meemingful  information  which  was  also  controllable  by 
contracting  managers.  Finally,  a  few  of  the  possible 
variables  contained  data  measurement  problems  in  the  7106 
reports. 

The  first  variable  not  used  in  the  DEA  model  was  the 
number  of  contracting  personnel  assigned  versus  the  number 
authorized.  Because  the  model  includes  all  three  categories 
of  possible  personnel  as  inputs  (BUYER,  CLERK,  and  MGT)  the 
assigned  versus  authorized  figure  would  have  provided 
redundant  information.  Additionally,  contracting  managers 
have  more  control  over  the  mix  of  human  resources  than  the 
established  squadron  manning  level. 

The  second  variable  not  used  in  the  DEA  model  was  the 
number  of  contracts  behind  schedule.  On  the  7106  report, 
this  measure  is  listed  as  the  nmnber  of  delinquent 
contracting  actions.  Although  this  measure  is  considered  an 
excellent  indication  of  the  quality  of  service  provided  by  a 
contracting  squadron,  the  7106  reports  indicate  a  serious 
data  measurement  problem.  Some  squadrons  had  over  400 
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delinquent  contracting  actions,  while  other  squadrons  showed 
no  delinquent  contracting  actions. 

The  third  variable  not  included  was  the  total  number  of 
line  items  received  by  each  squadron.  This  variable  is 
considered  em  input  to  the  contracting  squadron.  In  fact,  a 
squadron's  total  workload  is  driven  by  the  number  of  line 
items  received  from  its  customers  throughout  the  fiscal 
year.  However,  contracting  squadrons  have  no  control  over 
this  varieJsle.  The  number  of  line  items  received  is  driven 
by  external  factors  such  as  budget  levels  and  base  needs. 
Because  of  this  lack  of  control,  this  variable  was 
considered  less  meaningful. 

The  fourth  variable  omitted  from  the  DBA  model  was  the 
total  number  of  active  contracts  in  each  squadron.  This 
output  of  the  contracting  process  was  determined  significant 
by  contracting  managers.  It  is  also  a  meaningful  measure  of 
the  work  generated  by  a  squadron.  However,  three  other 
variables  were  included  in  the  DBA  model  which  captured  the 
information  present  in  the  total  active  contracts  measure. 
These  three  variables  were  SBRVICB,  CONST,  and  COMOD. 

The  fifth  variable  not  included  in  the  DBA  model  was 
the  number  of  priority  1-8  line  items  received  by  each 
contracting  squadron.  Priority  1-8  line  items  are  those 
items  which  are  higher  priority  requirements  (as  identified 
by  the  base  customer)  which  limits  the  amount  of  time  the 
contracting  squadron  has  to  make  the  purchase.  Like  the 
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line  items  received  measure,  this  vari£j3le  was  considered 
significant  by  contracting  managers.  Contracting  squadrons 
may  conduct  customer  education  to  train  base  personnel  on 
the  importance  of  keeping  the  priority  rate  at  manageed>le 
levels.  However,  squadrons  have  no  real  control  over  the 
actual  percentage  of  line  items  received  which  are  priority 
purchase  items.  Because  of  this  lack  of  control,  this 
variable  has  less  meaning  than  other  inputs  chosen  for  the 
DEA  application. 

The  sixth  variable  omitted  from  the  DEA  model  was  the 
niuober  of  modifications  executed  by  each  squadron.  This 
variable  is  a  measure  of  the  number  of  modifications  or 
changes  made  to  existing  contracting  instriunents .  This 
varied)le  was  rated  significant  by  respondents.  However, 
with  the  current  7106  report,  it  is  impossible  to 
distinguish  between  modifications  executed  because  of 
contracting  squadron  error  and  modifications  executed  as  a 
result  of  legitimate  changes  in  customer  requirements.  If 
this  distinction  could  be  made  in  the  future,  this  variable 
would  be  an  excellent  quality  measure  to  include  in  a 
subsequent  DEA  model.  Those  modifications  executed  due  to 
contracting  squadron  error  could  be  included  as  a 
minimization  variable.  Modifications  executed  because  of 
legitimate  chemges  in  customer  requirements  could  be 
included  as  an  output  for  DEA  to  maximize.  This  approach 
would  increase  the  DEA  rating  when  modifications  are 
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executed  due  to  legitimate  changes  £md  decrease  the  DEA 
rating  for  modifications  made  because  of  contracting  error. 
Because  of  the  current  measurement  difficulty  in  this 
vari€d>le,  it  was  not  considered  meaningful  for  inclusion  in 
this  DEA  model. 

The  seventh  variable  not  used  in  the  DEA  model  was  the 
percent  competitive  actions  measure.  This  variable  provides 
information  concerning  the  percent  of  the  squadron's  total 
actions  that  were  awarded  using  competitive  purchasing 
procedures.  The  information  provided  by  this  measure  was 
considered  redundant  with  the  total  dollars  awarded 
competitively  (COMP)  variable  included  in  the  DEA  model. 

Both  of  these  variables  were  rated  as  significant  outputs  by 
contracting  managers.  However,  the  COMP  variad)le  included 
was  considered  more  robust  because  it  also  provides  the 
magnitude  of  competitive  purchasing  conducted  at  each 
operational  contracting  squadron. 

A  total  of  seven  managerially  significant  inputs  and 
outputs  to  the  contracting  process  were  not  included  in  this 
DEA  model.  The  majority  of  the  omitted  variediles  contained 
redundant  information  when  compared  with  those  vari£d3les 
included  in  the  model.  However,  the  decision  to  omit 
certain  variedsles  does  not  mean  they  could  not  be  included 
in  a  future  DEA  application.  DEA  is  a  flexible  methodology 
which  allows  an  unlimited  combination  of  resources  and 
outputs  to  measure  operational  contracting  performance.  The 
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DEA  methodology  can  be  repeated  using  a  different 
combination  of  variables  in  order  to  examine  productive 
efficiency  with  emphasis  on  different  input  and  output 
variables . 

Part  I  Sminnary.  Part  I  of  this  chapter  provided 
research  results  concerning  investigative  questions  one  and 
two.  The  use  and  emalysis  of  the  management  survey 
identified  the  critical  resources  availeJ^le  to  operational 
contracting  managers.  This  same  survey  identified  critical 
outputs  to  the  operational  contracting  function.  Each 
managerial ly  significant  variable  was  further  examined  using 
Clarke's  five  characteristics  listed  in  Table  3.4.  A  total 
of  4  inputs  and  5  outputs  were  chosen  for  the  actual  DEA 
application. 

Part  II;  DEA  Application  and  Results 

After  identifying  the  9  variables  for  use  in  the  DEA 
model,  the  methodology  required  the  actual  application  of 
the  DEA  technique.  This  part  of  the  chapter  discusses  these 
results  and  answers  investigative  questions  three  through 
five.  Specifically,  Part  II  of  this  chapter  provides 
answers  to  the  investigative  questions  listed  in  T6d}le  4.6 
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TABLE  4.6  Investigative  Questions  3-5 


IQ  3:  What  are  the  results  of  the  application  of 
the  DEA  technique  to  operational 
contracting  performance  measurement? 

IQ  4:  What  information  does  the  DEA  output 
provide  the  Air  Force  operational 
contracting  manager? 

IQ  5:  What  characteristics  of  a  performance 
measurement  system  are  desired  by 
operational  contracting  managers  and  how 

_ are  they  addressed  by  DEA? _ 

DEA  Validation  and  Data  Used.  Before  the  DEA  process 
could  be  executed,  two  steps  to  ensure  accuracy  of  results 
were  completed.  First,  a  test  of  an  existing  DEA  problem 
was  executed  to  ensure  the  accuracy  of  the  methodology  and 
the  software  used.  Second,  the  archival  data  was  collected 
from  the  BCAS  7106  reports  for  each  base  being  analyzed. 

This  collection  resulted  in  the  necessary  elimination  of 
seven  bases  from  AFMC  and  a  clarification  of  what  was 
included  in  each  of  the  nine  variables. 

The  software  used  for  the  DEA  application  was  Storm 
version  3.0.  Storm  is  a  commercially  available  quantitative 
modeling  decision  support  software  package.  Storm  has  a 
standard  linear  programming  module  which  was  used  in  this 
DEA  model.  Before  executing  DEA  for  operational  contracting 
squadrons,  the  methodology  and  tableau  construction  was 
validated  using  an  existing  data  set  with  known  results. 

The  CCR  (Charnes,  Cooper,  and  Rhodes)  ratio  form  of  the  DEA 
relative  efficiency  measure  was  replicated  using  data  from 
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an  existing  study  by  Clarice  (14:377).  By  entering  data  from 
Clarice's  study  into  Storm  3.0,  the  DEA  results  were 
replicated  exactly. 

After  completion  of  the  methodology  validation,  data 
for  the  9  input  and  output  vari2d}les  was  collected  from  the 
7106  reports.  This  data  covered  fiscal  year  1992  which 
began  on  1  October  1991  and  ended  on  30  September  1992.  A 
complete  listing  of  the  input  and  output  data  for  the 
operational  contracting  squadrons  used  in  the  DEA  is 
provided  in  Table  4.7. 

TABLE  4.7  OEA  Input  and  Output  Data 


OUTPUTS  INPUTS 


SERVICE  CONST  COMOD  ACTION  COMP  EXP  BUYER  CLERK  MGT 

ACC  Bases 

Carswell 

40 

11 

4 

17648 

21031094 

216.92 

17 

3 

5 

79 

30 

16 

47599 

13554752 

162.39 

14 

1 

6 

Dyess 

51 

47 

6 

17893 

30074912 

205.38 

22 

3 

mm 

Ellsworth 

95 

69 

13 

35141 

37544932 

321.38 

27 

3 

9 

F.E.  Warren 

50 

51 

9 

24266 

19826151 

273.52 

23 

1 

7 

George 

26 

6 

2 

11087 

11999228 

77.88 

5 

1 

3 

Cannon 

29 

90 

6 

26916 

20708286 

264.4^ 

32 

3 

8 

Shaw 

149 

64 

26 

23104 

34184793 

408.13 

35 

3 

■■ 

Holloman 

55 

78 

70 

38666 

58702180 

472.75 

37 

4 

9 

47 

% 

34 

32033 

59407071 

411.44 

53 

8 

8 

25 

37 

8 

20519 

19166577 

199.40 

17 

2 

n 

88 

13 

33  ' 

1979 

77837439 

329.00 

25 

5 

9 

Homestead 

103 

121 

14 

24951 

19862203 

285.63 

30 

3 

8 

MacDill 

78 

118 

66 

92658 

44333345 

465.50 

47 

4 

8 

KI  Sawyer 

34 

82 

8 

.17062 

16966961 

243.38 

19 

2 

8 

Grand  Forks 

57 

65 

6 

50928 

36298917 

247.14 

30 

3 

8 

Loring 

35 

28 

4 

17322 

12051656 

179.13 

16 

2 

■■ 

IS15S9HH 

72 

112 

13 

19517 

17416578 

295.18 

25 

2 

7 

Ofiiit 
(55  Cons) 

145 

100 

26 

47650 

60214506 

347.39 

44 

6 

8 

Pope 

43 

47 

9 

12760 

17110521 

250.12 

25 

3 

5 

Nellis 

102 

66 

13 

36381 

73844156 

522.27 

49 

9 

7 

20 

5 

1 

7618 

5602865 

140.92 

8 

1 

mm\ 
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Table  4 . 7  Continued 


OUTPUTS 


SERVICE  CONST  COMOD  ACTION 


INPUTS 


EXP  BUYER  CLERK  MGT 


Mountain 

Home 


Baker 


Barksdale 


Beale 


Castle 


Fairchild 


Davis- 

Monthan 


GrifBss 


Minot 


Luke 


Seymour* 

Johnson 


Tyndall 


Wurtsmith 


Whiteman 


Howard 


AFMC  Bases 


Newark 


Brooks 


Robbins 


Hanscom 


Kiitland 


39 

49 

18 

24680 

29807899 

280.95 

27 

13 

6 

1 

12276 

2844394 

77.92 

6 

61 

49 

10 

39892 

26449439 

294.27 

23 

30 

66 

6 

26472 

24487624 

195.18 

15 

47 

12 

5 

22075 

14665634 

241.32 

16 

35 

40 

3 

34055 

19202276 

259.39 

29 

67 

67 

15 

66608 

34799160 

378.88 

34 

108 

93 

62 

43087 

42845731 

419.07 

37 

66 

86 

9 

33%2 

29354337 

202.38 

25 

159 

210 

72 

25138 

38278784 

350.30 

35 

40 

70 

21 

24014 

24447210 

204.07 

22 

71 

0 

52 

1875 

204281786 

378.23 

29 

140 

38 

39 

45160 

47922318 

402.16 

40 

33 

10 

2 

9486 

9440330 

122.97 

10 

49 

29 

13 

27753 

24339193 

274.69 

24 

37 

61 

42 

22134 

15971778 

157.68 

15 

8 

9 

8 

2644 

18178744 

57.00 

3 

7921  17391753  233.23  24 


14441  4581 172S  256.48  28 


33277  101470275  717.08  60 


31399  102991821  894.35  54 


32332  93872860  506.82  27 


24199  62019532  472.76  39 


In  order  to  allow  for  replication  of  this  research, 
clarification  of  the  operational  contracting  squadrons  rated 
and  variables  used  is  required.  The  two  coininands  used  for 
the  DEA  were  ACC  and  AFMC.  If  all  squadrons  in  these 
conuaands  were  used,  a  total  of  52  DMUs  would  be  rated  by 
DEA.  However,  while  collecting  the  actual  input  and  output 
measures  from  the  7106  reports,  seven  contracting  squadrons 
from  AFMC  were  removed  from  the  DMU  listing.  The  seven 
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squadrons  not  used  included:  Eglin  AFB,  Wright-Patterson 
AFB,  Tinker  AFB,  Edwards  AFB,  SMC/PKD,  McClellan  AFB,  and 
Hill  AFB.  These  squadrons  were  removed  because  information 
in  the  manpower  section  of  the  7106  report  was  incomplete. 
After  removal  of  these  squadrons,  the  DEA  was  performed  on 
the  remaining  45  operational  contracting  squadrons. 

The  five  output  variables  used  in  the  DEA  model  were 
SERVICE,  CONST,  COMOD,  ACTIONS,  and  COMP.  The  SERVICE  and 
COMOD  vari2Q3les  measured  the  nimber  of  active  service  and 
commodities  contracts  at  each  squadron  as  of  30  September 
1992.  The  CONST  variable  measured  the  active  construction 
and  architect  and  engineering  contracts  at  each  squadron  as 
of  30  September  1992.  The  ACTION  variable  measured  the 
cumulative  number  of  contracting  actions  executed  by  each 
contracting  squadron  throughout  fiscal  year  1992.  Finally, 
the  COMP  variable  measured  the  cumulative  dollars  spent 
using  competitive  procedures  throughout  fiscal  year  1992. 

The  four  input  or  resource  variables  included  EXP, 
BUYER,  CLERK,  and  MGT.  The  EXP  vari6d>le  measured  the  total 
office  experience  level,  in  years,  for  each  contracting 
squadron.  This  included  all  personnel  regardless  of 
position  within  the  squadron. 

The  BUYER  variable  provides  a  count  of  the  number  of 
buyers  or  contract  administrators  at  each  squadron. 

Although  Copper  Cap  and  Palace  Acquire  personnel  were  often 
assigned  to  the  management  section  of  the  7106  report,  they 
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were  counted  in  the  BUYER  variedsle  for  the  purposes  of  this 
study.  The  CLERK  variable  provides  a  count  of  the  number  of 
procurement  clerks,  secretaries,  ctnd  administrative  support 
personnel  assigned  to  each  squadron. 

Finally,  the  MGT  vari2d3le  provides  a  count  of  the 
management  in  contracting  squadron.  For  the  purposes  of 
this  study,  personnel  assigned  to  management  (front  office), 
auid  flight  chiefs  were  included  in  the  MGT  variaUQle. 
Administrative  support  and  secretarial  personnel  assigned  to 
the  front  office  were  not  counted  in  the  MGT  variable,  but 
were  included  in  the  CLERK  variable.  Contract  Management 
Officers  or  Officer  Trainees  were  counted  as  mcuiagement  if 
they  were  listed  in  the  management  section  of  the  7106 
report.  If  they  were  assigned  to  a  flight,  other  than  as  a 
flight  chief,  they  were  counted  as  part  of  the  BUYER 
varicd)le . 

I£LJU _ PEA  Application  and  Results.  The  PEA 

application  consisted  of  a  linear  program  for  each 
operational  contracting  squadron  or  PMU  under  analysis.  PEA 
revealed  15  PMUs  which  were  rated  one  hundred  percent 
efficient.  These  PMUs  received  a  PEA  rating  of  1.0.  Each 
contracting  squadron,  its  PEA  rating,  and  the  efficiency 
reference  set  (ERS)  for  those  squadrons  with  a  rating  less 
than  1.0  is  provided  in  Tedile  4.8. 
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TABLE  4.8  DBA  Ratings  and  Efficiency  Reference  Sets 


Effkkncy  Reference  Set 


Carswell 


Ellsworth 


•F.E.  Warren 


Cannon 


Shaw 


Holloman 


McConnell 


Offiit  (3908th 


Grand  Forks 


Loiin 


Nellis 


e  Beach 


Mountain  tfoine 


Baker 


Barksdale 


Beale 


Castle 


Fairchild 


Davis-Monthan 


GhfBss 


Minot 


Luke 


0.541183 


1.0 


0.687098 


.732404 


1.0 


0.969633 


0.744715 


1.0 


0.796676 


0.736773 


0.615908 


0.812527 


0.908895 


1.0 


0.881624 


0.970528 


0.506237 


1.0 


1.0 


0.475832 


0.865611 


0.436173 


0.598672 


0.601126 


0.721369 


0.953792 


0.516291 


0.632541 


0.876473 


0.953171 


1.0 


1.0 


m ,  MacDill,  Ofiut(55),  Lant  [Central),  Hanscom 


m .  Luke.  LangtevtCentral).  Brooks.  Hanscom 


Central).  Riyad.  Hanscom 


Central),  Brooks 


,  Brooks,  Hanscom 


MacDiU.  Luke.I^gley  (Central).  Hanscom  _ 


Central),  Brodcs,  Hanscom 


Central).  Hanscom 


’  Luke.  Brooks 


m,  Moodv.  Luke.  Hanscom 


Bergstrom ,  Luke.  Lam  [Central).  Rivad 


m,  MacDilL  Ofiiit  (55),  Luke,  Langley  (Central),  Hanscom 


MacDiU.  Langlev  (Central),  TyndaU,  KeUy,  Hanscom 


m 


m,  Minot,  Luke,  Langley  (Central),  Hanscom _ 


m 


Bergstfom,  MacDiU,  Langley  (Central),  Hanscom _ 


m,  Luke,  Riyad,  Hanscom 


yiSS. 


Berastrom,  MacDiU,  Luke,  Langley  (Central).  Hanscom 


Bergstrom,  MacDUl.  LanglOT  (Central)  _ 


MacDiU,  Brooks,  Hanscom 


>ur-Johnson  I 

[0.821353  1 

1  Bergstrom.  Mirmt.  Luke.  Lan 

551 

Central 

Wurtsmith 


Whiteman 


Howard 


1.0 


1.0 


0.640261 


0.539581 


Bergstrom,  Luke.  Hanscom  _ 


Bergstrom,  MacDiU.  Ofiiit  (55).  Langley  (Central).  TyndaU, 
Hanscom 
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TABLE  4 . 8  Continued 


Base 

DBA 

Radnz 

Efiiciciicy  Reference  Set 

Hanscom 

1.0 

Kiitland 

0.768904 

Langley  (Central).  MacDiU,  Hanscom 

*  Unenveloped  Solution 


The  results  indicate  a  large  reuige  of  relative 
efficiency  scores  for  each  of  the  squadrons  not  receiving  a 
rating  of  1.0.  These  scores  ranged  from  a  low  of  .436173  at 
Myrtle  Beach  AFB  to  a  high  of  .970528  at  Greuid  Forks  AFB. 

In  other  words,  Myrtle  Beach  AFB  is  43.6%  efficient  euid 
Grand  Forks  AFB  is  97.1%  efficient  relative  to  the  other 
DMUs. 

After  examination  of  the  general  results,  a  discussion 
of  the  15  squadrons  receiving  a  DBA  rating  of  1.0  is 
required.  In  order  to  ensure  each  of  these  squadrons  is 
receiving  a  true  relative  efficiency  score  of  100%,  the 
possibility  of  an  unenveloped  solution  (as  discussed  in 
Chapter  III)  must  be  considered.  If  one  of  the  DMUs 
receives  a  DBA  rating  of  1.0  resulting  from  an  unenveloped 
solution,  the  DMU  could  use  fewer  inputs  to  generate  the 
Scuae  level  of  output.  This  possibility  was  checked  by 
examining  the  Bfficiency  Reference  Set  (BRS)  of  inefficient 
DMUs.  Table  4.9  lists  all  squadrons  receiving  a  score  of 
1.0  and  the  number  of  times  the  squadron  appeared  in  the  BRS 
of  an  inefficient  DMU. 
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TABLE  4.9  DKOs  With  Efficivncy  Score  of  1.0 


HMD 

Muaber  of  Listings  in 

Another  DMDs  Efficiency 

Relerence  Set  fERS^ 

Bergstrom 

25 

*  F.E.  Warren 

0 

Shaw 

0 

MacDill 

12 

Moody 

2 

Offut  (55Cons) 

3 

Minot 

4 

Luke 

14 

Langley  (Central) 

19 

Tyndall 

2 

Howard 

1 

Riyad 

3 

Brooks 

7 

Kelly 

2 

Hanscom 

22 

*  Unenveloped  Solution 

The  results  show  that  13  of  the  15  squadrons  receiving 
a  DEA  rating  of  l.O  appear  in  the  ERS  of  other  inefficient 
squadrons.  F.E.  Warren  AFB  and  Shaw  AFB  fail  to  appear  in 
an  inefficient  DMU's  ERS  and  therefore  have  the  possibility 
of  being  unenveloped.  Poth  squadrons  could  have  a  non-zero 
slack  varisQ>le  assigned  to  one  or  more  of  the  inputs 
indicating  the  DMU  could  have  pi  e  duced  the  same  level  of 
output  with  less  of  one  or  more  inputs. 

An  examination  of  the  DEA  solution  for  Shaw  AFB  showed 
no  non-zero  slack  values  associated  with  the  inputs.  This 
indicates  Shaw  AFB  is  not  an  unenveloped  solution  auid  is 
considered  efficient.  The  DEA  solution  for  F.E.  Warren  AFB 
revealed  non-zero  slack  values  for  the  inputs  EXP  and  BUYER. 


101 


This  indicates  F.E.  Warren  AFB  received  a  DEA  rating  of  1.0 
as  the  result  of  an  unenveloped  solution.  F.E.  Warren  could 
have  produced  the  same  level  of  output  with  lower  levels  of 
office  experience  and  fewer  buyers  and/or  contract 
administrators.  Therefore/  F.E.  Warren  AFB  should  not  be 
considered  truly  efficient.  As  a  result  of  this  unenveloped 
solution,  the  number  of  relative  efficient  contracting 
squadrons  is  reduced  to  14  out  of  45  examined. 

10  4;  Information  Provided  to  Operational  Contracting 
Managers  from  DEA.  The  performance  measurement  emd 
evaluation  feedback  is  not  limited  to  the  DEA  rating  and 
identification  of  the  ERS  for  each  relatively  inefficient 
DMU.  DEA  provides  a  wealth  of  infomation  to  help  managers 
improve  performance  relative  to  their  peers.  After  giving 
managers  a  ranking  of  their  overall  relative  efficiency, 
improvement  information  can  be  gained  from  the  weights 
assigned  to  the  inputs  and  outputs  in  the  DEA  solution. 

DEA  allows  input  and  output  weights  to  vary  depending 
on  the  cheuracteristics  of  each  operational  contracting 
squadron.  Specifically,  the  DEA  methodology  allows  the 
assigned  input  and  output  weights  to  vary  depending  on  their 
relative  importemce  to  each  individual  squadron.  In  this 
way,  each  DMU  is  evaluated  according  to  the  inputs  and 
outputs  it  combines  in  the  most  efficient  manner.  The  DEA 
weights  assigned  to  each  contracting  squadron's  inputs  and 
outputs  are  shown  in  Table  4.10. 


102 


TABLE  4.10  OEA  Input  and  Output  Haights 


Sqnnour- 

Johnson 

0.0049 

m 

m 

■ 

0 

0.00423 

0.00331 

0.00001 

8.08E4)9 

m 

■ 

0.00016 

0.111 

0.00087 

m 

m 

0.00000669 

4.33E-09 

QIIHl 

0.00924 

0.10} 

0.00373 

Emm 

besssbh 

0 
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TABLE  4.10  Contlnusd 


BASE 

BUYER 

CLERK 

MGT 

SERVICE 

CONST 

COMOO 

ACTKW 

Wuitsmith 

o 

1 

0.0421 

0 

0.0832 

0.0191 

0 

0 

0 

0 

Whiteman 

0.00124 

0.0016 

0 

0.0888 

0.00289 

0.00021 

0 

0.00000943 

S.34E-09 

Howard 

0.00634 

0 

0 

0 

0.00471 

0.0166 

0 

0 

0 

OJ333 

0 

0 

0.01SS 

0 

0 

4.73E-08 

Newark 

0.00146 

0 

0.1319 

0 

0 

0.0179 

0 

0 

Bnxdcs 

0.00219 

0 

0 

0.0628 

0.00241 

0 

0.00368 

4.43E4>9 

mmm 

0.00018 

0 

0 

0.1249 

0 

0 

0.00806 

0.00000344 

0 

Robbins 

0 

0.00103 

0 

0.118 

0 

0 

0.00884 

0 

0 

Hansoom 

0.00163 

0 

0 

0.0237 

0 

0.00083 

0.00483 

0.0000038 

2.84E4)9 

Kiitland 

0 

0 

0 

msM 

0 

0.00103 

0 

0.00000961 

7.23E4» 

Note:  A  weight  of  0  is  equivalent  to  l.OE-09. 


The  weights  assigned  to  each  input  and  output  variable 
allow  the  contracting  metnager  to  examine  possible 
improvements.  Each  weight  is  emalogous  to  marginal 
productivity  in  economic  theory.  For  example,  the  weight 
assigned  the  BUYER  variable  at  Wurtsmith  AFB  is  .0421.  This 
means  for  every  unit  decrease  in  the  ntimber  of  buyers, 
Wurtsmith  AFB  can  expect  a  4.2%  increase  in  their  relative 
efficiency  score. 

Following  the  same  logic,  every  unit  increase  in  an 
output  will  result  in  a  efficiency  score  increase  equal  to 
the  corresponding  output  weight.  Again  using  Wuirtsmith  AFB 
as  an  example,  each  additional  active  service  contract  will 
produce  a  1.9%  increase  in  the  DEA  rating.  This  improvement 
analysis  is  valid  as  long  as  any  output  increase  or  input 
decrease  does  not  force  the  relative  efficiency  above  100%. 
Should  this  occur,  the  DEA  model  must  be  rerun  and  the 
results  reexamined. 
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The  information  provided  by  the  OEA  results  does  not 
stop  with  input  and  output  weights.  DEA  provides  the 
operational  contracting  memager  with  information  on  how  to 
improve  the  overall  relative  efficiency  score.  This  can  be 
accomplished  by  examining  the  DEA  generated  Efficiency 
Reference  Set  (ERS)  auxd  Hypothetical  Comparison  Unit  (HCU). 
Those  squadrons  receiving  a  DEA  rating  less  than  1.0  have  a 
corresponding  ERS  and  HCU.  This  comparison  allows  managers 
to  identify  which  inputs  and  outputs  can  be  changed  in  order 
to  improve  performance.  The  comparison  also  provides 
critical  information  2d3out  how  much  each  input  and  output 
variable  must  be  changed  in  order  to  achieve  relative 
efficiency.  An  example  of  ERS  analysis  for  Wurtsmith  APB  is 
provided  in  Table  4.11.  Recall  that  Table  4.8  provided  the 
ERS  for  each  of  the  inefficient  DMUs  including  Wurtsmith 
APB. 


TABLE  4.11  Wurtsmith  AFB  Compared  With  its  ERS 


waxtiavLtli 

Osrivud  Cosposlt* 

Ixeass  Inputs  of 

Actaal  Inputs 

EM  (Bacgstroa,  Luka, 

Wurtssatb  ATB  vs. 

S  Outputs 

S  Lsnglsy  (Sys)  AVBs 

Bsrgstrep,  Lulio,  c 
Lsnolsy(Sys)  AFBs 

SSRVICE 

33 

33 

0 

CONST 

10 

10 

0 

CQMCX) 

2 

2 

0 

ACTION 

9486 

9486 

0 

COMP 

9440330 

9440330 

0 

XXP 

122 . 97 

78.87 

44.1 

BOZXR 

10 

6.41 

3.59 

cuntx 

3 

.821 

2.179 

MST 

3 

1.922 

1.078 
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The  information  to  complete  this  ERS  analysis  is 
included  in  the  DEA  output.  The  HCU  is  merely  a  linear 
combination  of  the  input  and  output  vectors  of  those  OMUs 
listed  in  the  ERS  of  a  relatively  inefficient  DMU.  Once  the 
HCU  is  determined,  the  amount  of  excess  inputs  used  to 
produce  the  level  of  output  is  calculated.  By  using  this 
information,  the  operational  contracting  memager  cam  decide 
which  variables  to  reduce  2uid  the  euaount  of  reduction 
required.  Holding  output  levels  constant,  Wurtsmith  AFB 
would  have  to  reduce  its  inputs  by  the  amoiint  indicated  in 
order  to  become  relatively  efficient.  Management  judgment 
is  required  when  interpreting  these  results.  For  example, 
it  is  impossible  to  reduce  the  number  of  buyers  and/or 
contract  administrators  by  exactly  3.59. 

In  order  to  compare  each  DMU  with  its  ERS,  a  single  HCU 
must  be  derived  for  each  relatively  inefficient  DMU. 
Fortunately,  the  DEA  output  provides  all  the  information  to 
complete  these  calculations.  As  previously  stated,  the  HCU 
is  a  linear  combination  of  the  input  and  output  vectors  for 
each  DMU  listed  in  an  ERS.  The  DEA  output  provides  the  dual 
variable  (shadow  price)  for  each  DMU  in  the  ERS.  This  dual 
varied)le  is  the  multiplier  used  to  compute  the  HCU.  An 
example  calculation  of  the  HCU  is  provided  in  Table  4.12. 
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TABLE  4.12  Derivations  of  BCU  Values  For  WurtsBd.th  AFB 


Oatpata  £ 

Dual  v»R  For 

Output  C  Irput  Dual  VMl  for 

Output  a 

iDBtttS 

Baxgatcoa 

Voetor  for  Luka  APB  Proa 

Input  Vaetor 

*n  rxoai  oiA 

Baroatroa  APB  DBA 

Por  Luka  APB 

SZRVICB 

79 

159 

COMSV 

30 

210 

COMOD 

16 

72 

ACnON 

47599 

25138 

CCMP 

.1493 

13554752 

.0864 

38278784 

EXP 

162.98 

350.8 

BOXER 

14 

35 

CIXBX 

1 

5 

M8T 

6 

8 

oatputa  a 

Dual  for 

Output  a  Input  Vaetor 

Coapoalta 

1  1  1  — 

Banaeoa  Proa  DBX  for  lanalay  (8ya) 

BtS  (BCU) 

SERVICE 

156 

33 

CONST 

170 

30.77 

COMOO 

86 

12.73 

ACTICBl 

32332 

10857.15 

COMP 

.0479 

93872860 

9827521.4 

EXP 

506.82 

78.87 

BOXER 

27 

6.41 

CIXRK 

5 

.821 

M8T 

7 

1.922 

The  initial  DEA  results  demonstrated  the  ERS  for 
Wurtsraith  AFB  contained  Bergstrom  AFB,  Luke  AFB,  and  Hanscom 
AFB.  These  OMUs  are  used  to  derive  Wurtsmith’s  HCU.  Once 
derived,  the  HCU  input  levels  are  compared  to  the  actual 
input  levels  of  tlie  relatively  inefficient  DMU.  This 
analysis  was  shown  in  Table  4.11  when  Wuirt smith  AFB  was 
compared  to  its  ERS.  The  HCU  derivations  for  the  remaining 
relatively  inefficient  operational  contracting  squadrons  are 
provided  in  Appendix  D. 

The  DEA  results  provide  operational  contracting 
managers  with  additional  information  to  aid  performance 
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improvement.  DEA  assumes  that  any  point  along  the  efficient 
frontier  is  obtaineUsle.  Because  of  this,  am  inefficient  DMU 
is  not  limited  to  improvements  based  solely  on  the  ERS  and 
HCU  amalysis.  Managers  may  decide  it  is  not  possible  to 
reduce  inputs  in  the  same  proportions  as  specified  by  their 
HCU.  A  possible  path  to  a  DEA  rating  of  1.0  is  to  multiply 
each  of  the  input  levels  by  the  efficiency  rating  obtained 
from  the  DEA  results.  For  example,  Wurtsmith  AFB  could 
multiply  each  of  its  current  inputs  by  .640261  to  obtain  a 
rating  of  1.0. 

An  inefficient  DMU  can  also  determine  the  reduction  in 
a  single  input  required  to  obtain  an  efficiency  score  of 
1.0.  A  simple  formula  to  use  in  this  calculation,  along 
with  an  example  from  Wurtsmith  AFB  is  provided  in  Table 
4.13. 


TABLE  4.13  Single  Input  Reduction  Calculation 

1  -  Efficiency  Rating 
DEA  Weight  on  Decision  Input  Variable 

BUYER  Variable  ;  l  -  .640261  *  8.54 

at  Wurtsmith  AFB  .0421 

This  calculation  demonstrates  that  if  Wurtsmith  AFB 
reduced  its  niunber  of  buyers  or  contract  administrators  by 
8.5,  they  would  immediately  become  relatively  efficient. 
This  improvement  holds  true  as  long  as  other  input  and 
output  variables  remain  constant.  Once  again,  management 
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judgment  is  required  when  interpreting  this  result.  Because 
Wurtsmith  had  10  buyers  and/or  contract  administrators  at 
the  time  of  this  study,  it  may  be  unwise  to  remove  8  or  9 
personnel  from  the  buyer  variable. 

The  DEA  solution  provides  operational  contracting 
meuiagers  with  a  wealth  of  information  concerning  relative 
efficiency.  The  DEA  technique  provides  feedback  on  squadron 
productivity  and  allows  the  evaluation  of  alternate  methods 
of  performance.  After  examining  the  DEA  results,  the 
contracting  manager  can  decide  on  which  inputs  and  outputs 
to  change.  These  changes  are  made  with  full  knowledge  of 
target  input  and  output  levels  which  would  make  the  squadron 
relatively  efficient. 

IQ-.5? _ What  Characteristics  of  a  Performance 

Measurement  System  are  Desired  bv  Operational  Contracting 
Managers  and  How  are  They  Addressed  by  DEA?  After 
uncovering  critical  input  and  output  variables,  constructing 
and  executing  the  DEA  model,  and  providing  a  summary  of  DEA 
information,  the  research  turned  to  uncovering  desired 
characteristics  of  a  performance  measurement  system.  On  the 
same  suirvey  used  for  input  and  output  identification, 
operational  contracting  managers  were  asked  eleven  questions 
concerning  performance  measurement,  evaluation,  and 
feedback.  These  questions  addressed  current  methods  of 
evaluation  and  characteristics  of  a  desired  system. 
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After  collecting  the  survey  results,  a  large  sample 
test  of  hypothesis  was  conducted  to  uncover  responses  that 
differed  from  the  neutral  Likert  scale  value  of  3.0 
(37:356).  The  puzpose  of  this  test  was  to  uncover 
operational  contracting  management's  views  about  current 
methods  of  evaluation,  feedback,  and  desired  evaluation 
system  characteristics.  A  mean  response  which  varied 
significantly  from  the  neutral  Likert  response  of  3.0 
warranted  further  investigation.  Table  4.14  provides  the 
survey  question,  mean  response,  test  statistic  (z  value), 
and  significance  levels  for  each  of  these  qiestions.  • 
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TABLE  4.14  Suxrvey  Questions  Responding  to  IQ  5 

Strongly  Agree  Disagree  Neutral 

1  2  3 

Agree  Strongly  Agree 

4  '5 

# 

Question 

Mean 

Test  Stat 

8 

The  current  command  IG  evaluation  system 
satisfactorily  measures  organizational 
performance. 

2.92 

-1.07 

9 

An  evaluation  system,  different  from  the 
Command  IG,  would  be  useful  in  measuring 
overall  organizational  performance. 

3.6 

8.38  * 

Definitely  Not  Somewhat  Not 

Important  Important  Neutral 

1  2  3 

Somewhat  Definitely 

Important  Important 

4  5 

98 

The  current  IG  method  of  performance 
evaluation  provides  feedback  to  help  improve 
performance  which  is? 

3.63 

7.57 

99 

Current  Command  and  Air  Force  awards  provide 
feedback  concerning  performance  which  is? 

3.24 

1.93 

100 

Improving  operational  contracting  squadron 
productivity  and  efficiency  is  a  goal 
which  is? 

4.84 

45.46  * 

101 

A  contracting  evaluation  system  which 
simultaneously  evaluates  several  inputs 
and  outputs  to  the  process  (as  opposed  to 
single  measures  such  as  CALT) ,  is  a  tool 
which  is? 

4.497 

25.80  * 

102 

A  contracting  evaluation  system  which 
cosqsares  all  operational  contracting 
squadrons  while  taking  into  account 
differences  in  squadron  characteristics 
(such  as  manning,  experience,  and  workload) , 
is  a  tool  which  is? 

4.14 

12.35  * 

103 

A  contracting  evaluation  system  which  does 
not  rely  solely  upon  measures  which  the 
chain  of  command  (LG,  Wing  CC)  find  important 
is  a  tool  which  is? 

4.01 

11.39  * 

104 

A  contracting  evaluation  system  which 
compares  squadrons  relative  to  other 
contracting  squadrons,  is  a  tool  which  is? 

3.24 

2.37  * 

105 

A  contracting  evaluation  system  which 
compares  contracting  squadrons  to  the 
best  performers  as  opposed  to  the  average, 
is  a  tool  which  is? 

3.08 

0.81 

106 

A  contracting  evaluation  system  which 
provides  managers  with  timely  feedback, 
including  exact  data  on  resource 
utilization  and  relative  efficiencies 
compared  with  other  squadrons,  is  a  tool 
which  is? 

4.0 

10.73  * 

(*)  Significant  at  the  95%  level 
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The  results  for  the  questions  covering  current  methods 
of  performance  evaluation  (questions  8,  9,  98,  and  99) 
demonstrate  responses  that  may  indicate  a  need  for  an 
alternate  evaluation  system.  Managers  did  not  disagree  that 
the  current  Command  16  evaluation  system  satisfactorily 
measures  performance.  However,  when  asked  about  the 
usefulness  of  an  evaluation  system  other  than  the  Commemd 
IG,  managers  responded  significantly  above  the  neutral 
response.  When  asked  about  the  feedback  provided  by  the 
current  IG  method  of  evaluation,  managers  responded 
significantly  above  the  neutral  response.  Finally,  managers 
were  asked  about  the  importance  of  the  feedback  provided  by 
Air  Force  amd  Command  awards.  The  results  of  this  question 
did  not  differ  from  the  neutral  response. 

The  analysis  of  the  questions  concerning  current 
methods  of  operational  contracting  evaluation  did  not 
provide  conclusive  results.  Contracting  managers  did  not 
provide  a  significant  indication  that  they  were  completely 
disenchanted  with  performance  evaluation  techniques  in 
place.  However,  responses  to  the  remaining  survey  questions 
did  show  managers  desired  characteristics  that  are  not 
present  in  current  methods  of  performance  evaluation. 

When  asked  about  the  importance  of  improving 
operational  contracting  productivity  and  efficiency 
(question  100),  contracting  managers  responded  with  a  mean 
of  4.84.  The  test  of  hypothesis  indicated  this  mean  was 
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different  from  the  neutral  response  of  3.0  at  the  95% 
significance  level.  Because  OEA  addresses  each  DMUs 
relative  efficiency  or  productivity,  it  may  be  well  suited 
to  aid  contracting  managers  in  achieving  gains  in  this  area. 
As  demonstrated  in  IQ  4,  DEA  provides  a  relative  efficiency 
score  for  each  squadron  as  well  as  which  inputs  and  outputs 
to  change  for  improvement. 

The  response  to  the  importance  of  an  evaluation  system 
that  considers  several  inputs  and  outputs  to  the  process 
(question  101),  provided  a  mean  of  4.497.  This  mean  was 
found  to  be  significantly  different  from  a  neutral  response 
at  the  95%  level.  These  results  indicate  contracting 
managers  desire  an  evaluation  system  which  can  provide 
aggregate  analysis  of  inputs  and  outputs.  DEA  appears  to  be 
well-suited  to  meet  this  desire.  The  model  constructed  for 
this  study  utilized  4  inputs  and  5  outputs  while  providing  a 
single  measure  of  performance. 

The  next  two  questions  covered  manager's  opinion  about 
an  evaluation  system  which  takes  into  account  differing 
squadron  characteristics  (question  102)  and  a  system  which 
does  not  rely  solely  on  measures  which  the  local  base  chain 
of  command  find  important  (question  103).  The  operational 
contracting  manager's  responses  to  questions  102  and  103 
revealed  mean  responses  of  4.14  and  4.013  respectively. 

Both  of  these  means  differed  from  the  neutral  (3.0)  response 
at  the  95%  significance  level. 
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Contracting  managers  revealed  they  would  be  interested 
in  an  evaluation  system  which  takes  into  account  differences 
in  missions,  goals,  characteristics,  and  other  squadron 
specific  factors.  DEA  can  heutidle  this  desire  through 
flexible  weighting  of  the  input  and  output  variad>les  chosen. 
DEA  attempts  to  maximize  the  efficiency  score  for  each 
squadron,  thus  each  squadron  is  evaluated  on  what  it  does 
best.  In  this  manner,  DEA  can  evaluate  relative  efficiency 
between  squadrons  with  differing  characteristics. 

Additionally,  contracting  managers  do  not  like  the 
traditional  use  of  single  performcuice  measures.  Survey 
results  indicated  this  may  be  especially  true  when  the  base 
or  wing  chain  of  command  concentrates  on  specific  items  such 
as  CALT.  Once  again,  DEA  appears  to  be  an  improvement  over 
this  current  concentration  on  single  measure  performance 
evaluation.  Quite  simply,  DEA  can  handle  evaluation  of 
several  input  and  output  factors  at  one  time. 

When  asked  about  an  evaluation  system  which  compares 
squadrons  relative  to  other  squadrons,  memagers  provided  a 
mean  response  of  3.24v  This  was  significantly  different 
from  the  neutral  response  at  the  95%  level.  This  result 
indicates  contracting  managers  are  interested  in  a  system 
which  compares  relative  performance  between  squadrons.  As 
demonstrated,  DEA  calculates  efficiency  based  upon  relative 
comparison  between  DMUs.  In  this  manner,  squadrons  are 
compared  to  their  peers  instead  of  an  arbitrary  standard. 
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Question  105  did  not  result  in  a  mean  response 
significemtly  different  from  the  neutral  response.  This 
question  asked  managers  about  the  impoirtemce  of  comparing 
squadrons  to  the  best  performers  instead  of  a  measure  of 
average  performers.  It  appears  contracting  managers  are 
neutral  edx)ut  the  idea  of  being  compared  against  the  best 
squadrons  in  the  Air  Force. 

Finally/  when  asked  about  the  importance  of  an 
evaluation  system  providing  timely  feedback  on  resource 
utilization  2uid  relative  efficiency  (question  106),  managers 
responded  with  a  mean  of  4.0.  This  response  was 
significeuitly  different  from  the  neutral  response  at  the  95% 
level.  The  DEA  technique  is  well-suited  to  provide  resource 
utilization  and  relative  efficiency  feedback.  DEA  output 
includes  the  weights  assigned  to  decision  varieUsles,  along 
with  the  ability  to  examine  the  operations  of  the  HCU. 

In  summary,  the  results  generated  for  IQ  5  indicate 
operational  contracting  do  have  some  specific  desired 
characteristics  of  a  performance  measurement  emd  evaluation 
system.  These  characteristics  appear  to  be  well-suited  to 
the  characteristics  of  the  DEA  methodology.  Specifically, 
managers  want  to  improve  productivity  and  efficiency.  They 
think  em  evaluation  system  that  examines  several  inputs  and 
outputs  simultemeously  is  important.  Managers  would  like  to 
see  a  system  which  takes  into  account  different  squadron 
characteristics.  Contracting  managers  also  believe  too  much 
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anphasls  is  placed  on  single  performance  indexes.  Finally, 
managers  desire  a  system  which  provides  timely  feedback 
concerning  resource  utilization  emd  efficiency. 

As  discussed,  the  OEA  technique  appears  to  satisfy  the 
most  significeuit  concerns  of  operational  contracting 
managers.  Its  ability  to  allow  flexible  weighting  and 
evaluate  several  inputs  and  outputs  simultaneously  addresses 
the  significamt  characteristics  identified  by  managers. 
Although  contracting  managers  may  not  want  to  remove  current 
methods  of  performance  measurement,  the  characteristics  euid 
feedback  potential  of  DEA  meUce  it  a  viedsle  addition  to 
current  methods. 

Part  II  sumniArY.  This  part  of  the  chapter  provided 
answers  to  investigative  questions  three  through  five.  The 
OEA  model  was  constructed  emd  applied  to  45  operational 
contracting  squadrons  in  ACC  and  AFMC.  The  results  showed 
15  squadrons  which  received  an  efficiency  score  of  1.0.  One 
of  these  squadrons  was  unenveloped  euid  was  not  considered 
relatively  efficient.  All  information  provided  by  DEA 
output  was  reviewed.  Example  ERS  and  HCU  calculations  were 
demonstrated  using  Wurt smith  AFB.  Finally,  the  management 
survey  provided  desired  characteristics  of  ^ul  operational 
contracting  performance  evaluation  system.  The  survey 
responses  were  analyzed  and  desired  characteristics  compared 
to  the  output  provided  managers  through  DEA  application. 
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The  application  of  the  DEA  methodology  provided 
interesting  results.  The  results  were  divided  in  two  parts. 
Part  I  of  this  chapter  explained  the  selection  of  outputs 
and  resources  for  use  in  the  DEA  model.  Part  II  provided 
results  for  the  application  of  DEA  to  45  operational 
contracting  squadrons  in  ACC  and  AFMC. 

The  answers  to  investigative  questions  one  and  two  were 
discovered  by  analyzing  the  management  survey  to  identify 
critical  outputs  2Uiid  inputs  to  the  contracting  process.  The 
Large  Sample  Test  of  Hypothesis  revealed  16  possible  input 
and  output  vari2d>les  significant  at  the  95%  level.  They 
were  reduced  to  five  outputs  and  four  inputs  to  include  in 
the  DEA  model.  The  outputs  used  were  the  number  of  active 
service  contracts/  the  number  of  active  construction 
contracts/  the  number  of  active  commodities  contracts/  the 
total  number  of  contracting  actions/  and  the  total  dollars 
awarded  using  competitive  procedures.  The  inputs  included 
the  total  office  experience  level/  the  number  of  buyers/  the 
number  of  clerics /  and  the  number  of  management  at  each 
contracting  squadron. 

Investigative  question  three  was  answered  by  building 
and  executing  the  DEA  model  for  the  45  operational 
contracting  squadrons  under  study.  The  DEA  consisted  of  a 
linear  program  for  each  contracting  squadron.  The  results 
of  DEA  identified  15  squadrons  with  a  relatively  efficient 


score  of  l.O.  Further  2uialy8is  demonstrated  F.E.  Warren  AFB 
was  an  unenveloped  solution  emd  could  have  produced  the  same 
level  of  output  with  fewer  inputs. 

The  DBA  results  demonstrated  a  large  range  of  relative 
efficiency  scores  for  squadrons  not  receiving  a  rating  of 
1.0.  These  scores  r2Uiged  from  a  low  of  .436173  at  Myrtle 
Beach  AFB  to  a  high  of  .970528  at  Grand  Forks  AFB.  In  DBA 
terminology.  Myrtle  Beach  is  said  to  be  43.6%  efficient  and 
Greuid  Forks  is  97.1%  efficient  relative  to  the  other 
operational  contracting  squadrons. 

In  addition  to  the  DBA  ratings,  the  auiswer  to 
investigative  question  three  provided  an  efficiency 
reference  set  (BRS)  for  each  of  the  squadrons  not  receiving 
a  DBA  rating  of  1.0.  The  BRS  provides  the  contracting 
memager  with  a  group  of  efficient  squadrons  to  emulate  for 
future  performance  improvement.  The  squadrons  listed  in  the 
BRS  combine  inputs  to  create  outputs  in  a  similar  fashion  as 
the  inefficient  squadron.  In  other  words,  the  squadrons  in 
the  BRS  use  production  technology  similar  to  the  inefficient 
squadron. 

The  answer  to  investigative  question  four  detailed  the 
performsuice  improvement  information  provided  the  operational 
contracting  manager  from  the  DBA  output.  This  information 
was  contained  in  four  parts  of  the  DBA  results.  First,  the 
DBA  output  includes  decision  variable  weights  which  are 
analogous  to  marginal  productivity  in  economic  theory. 
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Second,  DEA  output  allows  the  contracting  manager  to  examine 
the  excess  inputs  used  in  the  contracting  process.  By 
reducing  the  excess  inputs  emd  holding  the  output  level 
constant,  a  contracting  squadron  can  improve  performance  and 
receive  a  DEA  rating  of  1.0.  Third,  the  DEA  output  provides 
a  dual  solution  (shadow  price)  for  each  squadron  appearing 
in  the  ERS  of  an  inefficient  squadron.  This  dual  solution 
is  the  multiplier  used  to  calculate  the  Hypothetical 
Comparison  Unit  (HCU).  Once  calculated,  the  HCU  provides 
the  contracting  m2uiager  with  exact  quantities  of  inputs  and 
outputs  to  target  for  performance  improvement.  Finally,  DEA 
output  provides  a  single  input  reduction  calculation  for 
performemce  improvement.  This  calculation  gives  the 
reduction  needed  in  one  input  in  order  to  reach  the 
efficient  frontier  and  receive  a  DEA  rating  of  1.0. 

Investigative  question  five  was  concerned  with 
contracting  manager  perceptions  about  current  methods  and 
desired  characteristics  of  performance  measurement  and 
evaluation.  Eleven  questions  on  the  management  survey  were 
emalyzed  using  a  Large  Sample  Test  of  Hypothesis.  The 
results  of  this  analysis  indicated  that  DEA  is  a  useful 
addition  to  operational  contracting  performance  evaluation. 
Specifically,  managers  expressed  a  need  for  improving 
productivity  euid  efficiency.  The  DEA  technique  is  designed 
to  measure  efficiency  and  productivity  relative  to  the  best 
performing  contracting  squadrons. 
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Additionally,  contracting  memagers  expressed  a  desire 
for  eua  evaluation  system  which  accounts  for  differences  in 
squadron  characteristics,  mission,  and  purchase  complexity. 
By  allowing  flexible  weighting  of  the  input  and  output 
coefficients,  DEA  accounts  for  these  differences.  The 
squadrons  which  received  DEA  ratings  of  1.0  varied  in  size, 
total  contracting  actions,  2ind  total  funds  obligated. 

Finally,  contracting  managers  expressed  desires  for  an 
evaluation  system  which  combines  several  inputs  and  outputs 
into  a  single  index  of  performance.  The  survey  results 
indicated  managers  believe  too  much  emphasis  is  currently 
placed  on  single  measure :  performance  such  as  CALT .  DEA 

demonstrated  an  ability  to  combine  several  individual 
performance  measures  into  an  aggregate  rating  of  overall 
performance . 

The  results  presented  in  this  chapter  achieved  the  two 
main  research  objectives.  First,  a  DEA  model  was 
constructed  for  use  in  operational  contracting  performance 
measurement.  Second,  the  DEA  output  was  evaluated  for  its 
potential  use  to  operational  contracting  management.  DEA  is 
a  flexible  methodology  which  allows  the  examination  of 
various  input  and  output  combinations  to  the  operational 
contracting  process.  The  DEA  technique  proved  to  be  an 
excellent  addition  to  current  methods  of  measuring 
operational  contracting  performance. 
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V.  Summary. 


gQPglugiQBg/  sod  Recommendations 

Summary 

Due  to  the  critical  mission  support  nature  emd 
magnitude  of  taxpayer  dollars  expended  by  operational 
contracting,  the  ability  to  measure  and  provide  perform^Ulce 
feedback  to  the  contracting  manager  is  essential. 

Contracting  performance  includes  the  related  concepts  of 
efficiency  and  productivity.  Both  concepts  are  concerned 
with  how  well  a  contracting  squadron  converts  inputs  or 
resources  into  outputs  (42:420).  Because  they  are  major 
components  of  overall  purchasing  performance,  the  cLbility  to 
measure  efficiency  fimd  productivity  is  an  important  tool  for 
operational  contracting  performance  improvement. 

The  current  systems  to  measure  operational  contracting 
efficiency  and  productivity  contain  specific  weaknesses. 
First,  the  systems  fail  to  aggregate  individual  performance 
indicators  into  an  index  of  overall  performance.  Second, 
they  do  not  measure  performance  relative  to  other 
operational  contracting  squadrons.  Third,  current  systems 
do  not  provide  feedback  on  specific  ways  to  improve  overall 
squadron  productivity.  Finally,  they  provide  limited 
ability  to  compare  operational  contracting  squadrons  of 
different  size,  mission,  objectives,  and  purchasing 
complexity. 
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Because  of  these  limitations/  SAF/AQCO  and  AFLMA/LGC 
identified  a  need  to  develop  and  evaluate  an  alternate 
method  of  measuring  operational  contracting  performance 
(27).  Because  productivity  and  efficiency  are  major 
components  of  overall  performance/  this  research 
concentrated  on  measuring  productive  efficiency  of 
operational  contracting  sguadrons.  Specifically/  the 
relatively  new  approach  of  Data  Envelopment  Analysis  (OEA) 
was  chosen  to  investigate  as  an  alternate  performance 
evaluation  system. 

The  overall  research  objective  was  to  design  a  DEA 
evaluation  system  that  provides  an  alternative  to  current 
methods  of  measuring  operational  contracting  performance. 

In  order  to  meet  this  objective/  a  DEA  model  for  operational 
contracting  was  developed  and  evaluated  for  potential  use  by 
contracting  managers.  The  use  of  five  investigative 
questions  facilitated  the  research  process. 

10  1  &  2:  What  are  the  Most  Critigal  Outputs  and 
Resources  in  Operational  Contracting?  A  survey  of 
operational  contracting  managers  revealed  16  possible 
operational  contracting  outputs  and  resources  which  were 
significant.  After  eliminating  variables  which  provided 
redundant  information/  or  variables  difficult  for  management 
control/  nine  varicd>les  were  included  in  the  DEA  model.  The 
output  variables  included  the  number  of  service  contracts/ 
the  niimber  of  construction  contracts/  the  ntunber  of 
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commodities  contracts,  the  total  number  of  contracting 
actions,  and  the  total  dollars  awarded  using  competitive 
procedures.  The  input  variables  included  the  total  office 
experience  level,  the  number  of  buyers,  the  number  of 
clerks,  and  the  niunber  of  managers. 

Ifl  — What  are  the  Results  of  the  Application  of  ths. 
PEA  Technigue  to  Operational  Contracting  Performance 
Measurement?  The  DEA  resulted  in  15  squadrons  receiving  an 
efficiency  score  of  1.0.  Analysis  demonstrated  one  of  these 
squadrons  was  an  unenveloped  solution.  This  squadron  was 
not  considered  efficient  because  it  could  have  produced  the 
same  output  using  less  of  one  or  more  inputs.  After 
eliminating  the  unenveloped  solution,  14  squadrons  remained 
efficient  relative  to  the  other  squadrons  in  the  DEA  model. 

Oiice  DEA  ratings  were  obtained,  the  Efficiency 
Reference  Set  (ERS)  for  each  relatively  inefficient  squadron 
was  reported.  The  ERS  provides  a  list  of  Decision  Making 
Units  (DMUs)  for  the  contracting  manager  to  use  in 
performance  improvement.  Specifically,  the  ERS  for  a 
squadron  is  a  set  of  DMUs,  receiving  a  relatively  efficient 
DEA  rating,  which  envelope  the  inefficient  squadron.  The 
ERS  provides  contracting  managers  with  a  set  of  squadrons  to 
emulate  in  future  operations. 

10  4;  What  Information  Does  DEA  Output  Provide  the  Air 
Force  Operational  Contracting  Manager?  In  addition  to  the 
efficiency  score  and  ERS,  the  DEA  output  provided  the 
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operational  contracting  manager  with  a  wealth  of  performance 
improvement  information.  First,  the  DEA  output  provided 
weights  for  each  input  and  output  variable  used  in  the 
analysis.  Each  weight  is  analogous  to  marginal  productivity 
in  economic  theory.  For  example,  the  weight  assigned  to  the 
BUYER  varicible  at  Wurtsmith  AFB  was  .0421.  This  meams  for 
eve3:y  unit  decrease  in  the  number  of  buyers,  Wurtsmith  can 
expect  a  4.2%  increase  in  their  DEA  rating. 

Second,  DEA  allowed  the  calculation  of  the  excess 
inputs  used  by  inefficient  squadrons.  This  calculation  was 
made  by  comparing  the  current  input  level  to  the  inputs  used 
by  the  squadrons  in  the  ERS.  Wurtsmith  AFB  was  used  for  an 
example  in  Chapter  IV.  In  order  for  Wurtsmith  AFB  to  move 
to  the  efficient  frontier,  it  could  reduce  its  total 
experience  level  by  44.1  years,  the  number  of  buyers  by 
3.59,  the  number  of  clerks  by  2.179,  and  the  number  of 
management  personnel  by  1.078.  By  making  these  reductions, 
while  holding  output  constant,  Wurtsmith  AFB  would  become 
relatively  efficient. 

Third,  DEA  provides  the  ability  to  derive  Hypothetical 
Comparison  Unit  (HCU)  values  for  each  inefficient  DMU.  Once 
again,  Wurtsmith  AFB  was  used  as  an  example  in  Chapter  IV. 
The  HCU  is  a  linear  combination  of  the  input  and  output 
vectors  for  each  DMU  listed  in  the  ERS.  The  DEA  output 
provides  a  dual  variable,  or  shadow  price,  for  each  DMU 
listed  in  the  ERS.  This  dual  variable  is  the  multiplier 
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used  to  calculate  the  HCU.  Once  calculated,  the  HCU 
provides  a  target  resource  level  for  relatively  inefficient 
DMUs.  If  this  resource  level  is  obtained,  holding  output 
levels  constant,  the  DMU  will  receive  a  DEA  rating  of  1.0. 

The  final  piece  of  performeuice  improvement  information 
was  the  single  input  reduction  calculation.  Because  DEA 
assumes  any  point  along  the  efficient  frontier  is 
obtainable,  an  inefficient  DMU  is  not  limited  to 
improvements  based  solely  on  ERS  and  HCU  analysis.  An 
inefficient  DMU  can  multiply  each  of  the  inputs  by  the  DEA 
rating  in  order  to  reach  the  efficient  frontier. 
Additionally,  a  simple  formula  can  be  employed  to  calculate 
the  reduction  required  in  a  single  input  to  receive  a  DEA 
rating  of  1.0. 

10  5;  What  Characteristics  of  a  Performance 
Measurement  System  are  Desired  bv  Operational  Contracting 


Managers  and  How  Are  They  Addressed  by  DEA?  The  same  survey 
used  for  input  and  output  identification  contained  eleven 
questions  to  query  contracting  managers  about  performance 
measurement,  evaluation,  and  feedback.  Contracting  managers 
indicated  that  improving  operational  contracting 
productivity  and  efficiency  are  important  objectives.  They 
expressed  a  desire  for  an  evaluation  system  which  can 
measure  several  inputs  and  outputs  to  the  contracting 
process  as  opposed  to  a  system  which  examines  singular 
performance  indicators.  Finally,  contracting  managers 


indicated  a  desire  for  an  evaluation  system  which  can 
compare  different  squadrons  while  accoiuiting  for  differences 
in  squadron  characteristics,  missions,  and  purchase 
complexity . 

Conclusions 

The  results  of  this  research  indicate  DEA  is  em 
important  tool  for  enhancing  performance  evaluation  of  Air 
Force  operational  contracting  squadrons.  The  information 
provided  from  the  OEA  output  appears  to  be  an  improvement 
over  the  current  methods  of  operational  contracting 
performance  evaluation.  Additionally,  the  OEA  feedback 
addresses  performance  evaluation  characteristics  desired  by 
operational  contracting  managers. 

Strengths  of  DEA.  There  are  several  characteristics  of 
DEA  which  demonstrate  its  potential  for  operational 
contracting  performance  evaluation.  First,  DEA  can  create  a 
single  efficiency  rating  based  upon  multiple  inputs  and 
outputs  to  the  contracting  function.  Current  methods  of 
operational  contracting  evaluation  do  not  allow  the 
combination  of  several  performance  indicators  into  a  single 
index  of  overall  performcuice. 

Because  DEA  allows  for  flexible  weighting  of  the 
decision  variables  (inputs  and  outputs)  it  can  account  for 
differences  in  squadron  characteristics  such  as  size, 
mission,  and  purchase  complexity.  In  this  manner,  all  Air 
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Force  operational  contracting  squadrons  can  be  compared  with 
confidence  that  squadron  differences  are  taken  into  account. 
The  results  of  the  management  survey  indicate  this 
characteristic  is  desired  by  operational  contracting 
meuiagers . 

DEA  provides  an  increased  amount  and  quality  of 
performance  improvement  information  when  compared  with  the 
current  methods  of  contracting  evaluation.  Contracting 
managers  indicated  improvements  in  productivity  and 
efficiency  are  import2Lnt  goals  of  performance  improvement. 
DEA  provides  more  than  an  overall  rating  of  performance. 

The  ability  to  compare  squadron  performance  to  the  ERS  and 
HCU  allow  managers  to  create  a  specific  plan  for 
improvement.  By  examining  the  DEA  weights  assigned  to  the 
inputs  and  outputs/  the  contracting  manager  can  conduct 
what-if  analysis  concerning  productivity  and  efficiency 
improvement . 

Additionally,  DEA  gives  contracting  managers  the 
ability  to  set  specific  goals  for  input  reductions  or 
increases  in  outputs.  Once  established,  these  goals  can  be 
inserted  into  the  DEA  model  and  a  new  efficiency  rating 
established.  This  provides  the  ability  to  not  only 
estsUslish  a  course  of  action,  but  to  inform  the  workforce  of 
the  resulting  rating  if  the  goals  are  achieved.  The  current 
methods  of  operational  contracting  performance  evaluation 
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can  not  tell  a  manager  how  to  achieve  specific  goals  and  the 
overall  squadron  rating  once  the  goals  are  achieved. 

Finally,  the  DEA  evaluation  technique  is  well-suited 
for  continuous  process  improvement.  The  efficient  frontier 
established  by  DEA  is  not  fixed  in  a  permanent  position.  If 
the  technique  is  used  at  given  intervals,  over  a  period  of 
time,  the  frontier  will  continue  to  move  towards  more 
efficient  operations.  A  contracting  squadron  originally 
receiving  a  rating  of  1.0  may  be  surpassed  by  squadrons 
using  DEA  feedback  to  improve  operations.  In  this  manner, 
the  entire  operational  contracting  fimction  can  achieve 
continuous  improvements  in  productivity. 

Limitations  of  the  Research  and  DEA  Technique.  No 
system  of  performance  evaluation  is  without  limitation.  The 
primary  limitations  to  the  use  of  DEA  were  discussed  in 
Chapter  III  of  this  research.  However,  there  are  four 
important  limitations  that  should  be  reviewed  at  this  time. 
These  limitations  apply  to  the  data  used  in  evaluation  and 
the  interpretation  of  the  decision  varied>le  weights  assigned 
by  DEA. 

First,  this  DEA  application  did  not  include  any  measure 
of  quality  for  operational  contracting  squadrons.  The 
current  data  available  on  the  7016  report  provides  limited 
quality  indicators.  A  possible  measure  was  the  number  of 
contracts  behind  schedule  at  each  squadron.  This  is  listed 
as  the  niunber  of  delinquent  contracting  actions  on  each  7106 
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report.  However,  this  measure  was  omitted  from  the  DEA 
model  because  an  examination  of  all  the  squadrons  under 
study  revealed  a  large  disparity  in  delinquent  contracts  at 
each  base. 

The  lack  of  quality  measures  in  this  research  is  not  2m 
indication  of  a  true  DEA  limitation.  DEA  is  well~suited  for 
the  use  of  qualitative  variables  in  the  measurement  of 
performance.  However,  the  current  BCAS  system  provides 
limited  reporting  of  quality  service  indicators.  Once 
developed,  quality  variables,  such  as  the  actual  delivery 
time  of  a  purchase,  could  be  inserted  into  the  current  DEA 
model  and  used  for  a  more  robust  indication  of  performance. 
However,  since  the  primary  purpose  of  this  research  was  to 
measure  productivity  and  efficiency,  the  lack  of  quality 
indicators  did  not  limit  the  results. 

Second,  each  DMU  must  collect  and  report  input  and 
output  data  in  a  consistent  manner.  The  variables  chosen 
for  this  research  are  reported  consistently  between 
operational  contracting  squadrons.  However,  future  research 
must  be  careful  when  expanding  this  application  to  include 
other  BCAS  reported  variedsles.  For  example,  it  was  apparent 
that  each  squadron  may  not  report  the  delinquent  contracts 
measure  in  a  consistent  manner. 

Third,  DEA  assigns  input  and  output  variables  weights 
in  accordance  with  a  mathematical  algorithm.  The  technique 
can  not  comprehend  true  economic  or  social  value  attached  to 
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a  decision  variable.  For  example.  Congressional  interest 
may  increase  the  importance  a  variable  such  as  small 
business  dollars  awarded.  OEA  will  not  automatically 
consider  this  important  social  aspect  of  a  particular 
variable.  However,  the  OEA  linear  program  can  be  modified 
to  limit  the  range  of  specific  input  or  output  varieQsle 
weights  in  order  to  account  for  variables  of  social  or 
economic  importemce. 

Fourth,  DEA  reports  only  technical  efficiency.  Each 
DMU  is  allowed  to  maximize  its  efficiency  score  subject  to 
the  constraints  of  the  linear  programming  problem.  The 
contracting  manager  must  be  careful  not  to  place  economic 
value  on  a  DEA  generated  variable  weight.  This  limitation 
does  not  prevent  the  contracting  memager  from  developing  a 
course  of  action  to  improve  performance  relative  to  other 
squadrons . 

Finally,  DEA  is  only  a  tool  to  help  improve  operational 
contracting  performance.  The  performemce  improvement 
information  provided  must  be  interpreted  with  management 
judgment.  It  may  be  difficult  to  reduce  a  specific  input 
due  to  Air  Force  mission  constraints.  For  example,  it  may 
not  be  possible  for  some  squadrons  examined  in  this  research 
to  reduce  the  number  of  buyers  due  to  contingency 
contracting  requirements. 
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Because  this  research  is  the  first  attempt  to  apply  the 
DEA  technique  to  the  government  purchasing  function,  there 
are  several  areas  of  future  research.  The  DEA  technique  is 
relatively  new  in  management  literature,  but  it  appears  to 
provide  improved  ability  to  measure  performance  in  service 
providing  fiinctions.  This  ability  makes  DEA  a  cemdidate  for 
future  research  endeavors. 

A  logical  extension  of  this  research  is  to  increase  the 
number  of  contracting  squadrons  evaluated.  The  significant 
input  and  output  varied)les  uncovered  in  this  research  could 
be  used  to  extend  the  current  DEA  model  to  include  all 
operational  contracting  squadrons  in  the  Air  Force.  As  long 
as  the  decision  variable  data  is  reported  in  a  consistent 
maumer,  the  analysis  could  also  include  systems  acquisition 
organizations  in  AFMC. 

Another  possibility  for  future  research  would  be  to 
work  with  SAF/AQCO  and  AFLMA/L6C  to  develop  quality  measures 
to  include  in  a  future  DEA  application.  Current  AFLMA/LGC 
research  is  attempting  to  develop  a  measure  of  total 
delivery  time  for  a  purchase  (27).  This  measure  would  be  a 
true  -indication  of  how  well  a  contracting  squadron  is 
supporting  the  base.  As  long  as  this  measure  is  reported  in 
a  consistent  fashion,  it  would  be  a  good  candidate  to  expand 
the  DEA  technique  to  include  quality  evaluation. 
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An  importeuit  possibility  for  future  research  includes 
the  integration  of  the  DEA  methodology  into  the  everyday  use 
at  the  operational  contracting  squadron.  This  would  require 
the  modification  of  the  BCAS/  or  future  management 
information  system,  to  include  the  DEA  methodology.  Several 
reseeurch  efforts  could  be  conducted  on  this  topic.  First,  a 
DEA  program  must  be  created  that  is  user-friendly  for 
contracting  personnel.  Second,  the  program  should  be 
integrated  into  the  operational  contracting  management 
information  system  so  that  data  from  all  bases  is  readily 
available  for  DEA  modeling.  Finally,  system  operating 
instructions  must  be  created  so  that  the  DEA  progreun  could 
be  easily  understood  and  used. 

Another  possible  area  of  future  research  would  be  to 
examine  the  effect  of  changing  input  and  output  variables  on 
the  overall  efficiency  score  received  by  contracting 
squadrons.  Squadrons  who  received  a  DEA  rating  of  1.0  in 
this  research  may  not  be  a  relatively  efficient  DMU  if  the 
resource  and  output  mix  is  altered.  Separate  DEA  models 
could  be  compared  in  order  to  examine  the  effects  of 
chemging  input  and  output  measures  analyzed. 

A  final  area  of  future  research  would  include  the 
modification  of  the  current  DEA  model  so  that  operational 
contracting  managers  could  examine  performance  within  each 
of  their  respective  flights.  The  input  and  output  variables 
could  be  broken  down  by  contracting  flight.  This  research 
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would  allow  contracting  memagers  to  detersiine  their  optimal 
resource  allocation  among  the  flights  within  the  squadron. 

As  previously  stated /  this  research  is  the  first 
application  of  DEA  to  government  purchasing  so  there  are 
many  options  for  future  research.  Because  DEA  was  designed 
for  performance  measurement  in  service-providing  functions, 
it  is  well-suited  for  various  applications  in  the  DOD. 
Whenever  an  analysis  of  multiple  inputs  and  outputs  from  a 
specific  process  is  desired,  the  DEA  technique  can  provide 
improved  performance  measurement  edjility. 


133 


1.  Total  line  items  received 

2.  Total  line  items  received  priority  1-6 

3.  Percent  of  total  line  items  received  priority  1-8 

4.  Total  modifications  executed 

5.  Total  dollars  awarded 

6.  Number  of  sales  contracts  awarded 

7.  Dollar  value  of  sales  contracts  awarded 

8.  Total  centralized  actions  awarded 

9.  Total  centralized  line  items  awarded 

10.  Total  centralized  dollars  awarded 

11.  Total  decentralized  actions  awarded 

12.  Total  decentralized  dollars  awarded 

13.  Total  number  of  priced  actions  awarded 

14.  Total  niunber  of  unpriced  actions  awarded 

15.  Percent  of  total  line  items  purchased  that  were  priced 

16.  Total  centralized  actions  from  0-500 

17.  Total  centralized  actions  from  501-10,000 

18.  Total  centralized  actions  from  10 ) 001-25, 000 

19.  Total  centralized  actions  over  25,000 

20.  Total  decentralized  actions  from  0-500 

21.  Total  decentralized  actions  from  501-10,000 

22.  Total  decentralized  actions  from  10,001-25,000 

23.  Total  centralized  actions  over  25,000 

24.  Total  active  contracting  actions 

25.  Total  active  delinquent  contracting  actions 

26.  Percent  total  active  delinquent  contracts 

27.  Total  active  contract  modifications 

28.  Percent  total  active  contract  modifications 
Complex  Construction 

29.  Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 

34.  Total  active  contracts 

Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 
Complex  *;ftfw*^itie8 

39.  Total  active  contracts 

40.  Number  of  contracts  delinquent 

41.  Percent  of  contracts  delinquent 

42.  Nund^er  of  contract  modifications 

43.  Percent  of  contract  modifications 


30. 

31. 

32. 

33. 


35. 

36. 

37. 

38. 
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Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 


Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Niunber  of  contract  modifications 
Percent  of  contract  modifications 


Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 


Total  active  contracts 
Niunber  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Niunber  of  contract  modifications 
Percent  of  contract  modifications 


Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 


Total  active  contracts 
Number  of  contracts  delinquent 
Percent  of  contracts  delinquent 
Number  of  contract  modifications 
Percent  of  contract  modifications 


74.  Total  active  contracts 

75.  Number  of  contracts  delinquent 

76.  Percent  of  contracts  delinquent 

77.  Number  of  contract  modifications 

78.  Percent  of  contract  modifications 

gfltsis  agfl 

79.  Total  active  contracts 

80.  Number  of  contracts  delinquent 

81.  Percent  of  contracts  delinquent 

82.  Number  of  contract  modifications 

83.  Percent  of  contract  modifications 


Purchas*  Orders 

84.  Total  active  contracts 

85.  Number  of  contracts  delinquent 

86.  Percent  of  contracts  delinquent 

87.  Number  of  contract  modifications 

88.  Percent  of  contract  modifications 
Unpriced  Purchase  Orders 

89.  Total  active  contracts 

90.  Niimber  of  contracts  delinquent 

91.  Percent  of  contracts  delinquent 

92.  Number  of  contract  modifications 

93.  Percent  of  contract  modifications 

94.  Total  active  contracts 

95.  Number  of  contracts  delinquent 

96.  Percent  of  contracts  delinquent 

97.  Number  of  contract  modifications 

98.  Percent  of  contract  modifications 
Sales  Actions 

99.  Total  active  contracts 

100.  Number  of  contracts  delinquent 

101.  Percent  of  contracts  delinquent 

102.  Number  of  contract  modifications 

103.  Percent  of  contract  modifications 

104.  Total  competitive  actions 

105.  Total  competitive  dollars 

106.  Total  competitive  and  noncompetitive  dollars 

107.  Percent  competitive  dollars 

108.  Intergovernmental  actions 

109.  Intergovernmental  dollars 

110.  FMS  actions 

111.  FMS  dollars 

112.  NAFI  actions 

113.  NAFI  dollars 

114.  Defense  fuel  actions 

115.  Supply  center  dollars 

116.  Zero  dollar  modification  actions 

117.  Zero  dollar  actions 

118.  Non  profit  actions 

119.  Non  profit  dollars 

120 .  Utilities  actions 

121.  Utilities  dollars 

122.  Commissary  resale  actions 

123.  Commissary  resale  dollars 

124.  Section  8 A  actions 

125.  Section  8A  dollars 

126.  Other  competitive  code  98  actions 

127.  Other  competitive  code  98  dollars 

128.  Total  set  aside  actions 

129.  Total  set  aside  dollars 

130.  Total  large  business  actions  (when  available  for  small 
business) 
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131.  Total  large  business  dollars  (when  available  for  small 
business ) 

132.  Small  business  percentage  of  total  dollars  available 
for  small  business 

133.  Disadvemtaged  business  actions 

134.  Disadvantaged  business  dollars 

135.  Women  owned  business  actions 

136.  Women  owned  business  dollars 

137.  Reserved  for  small  business  actions 

138.  Reserved  for  small  business  dollars 

139.  Authorized  civilians 

140.  Assigned  civilicms 

141.  Authorized  airmen 

142.  Assigned  airmen 

143.  Authorized  officers 

144.  Assigned  officers 

145.  Assigned  management  (front  office  and  branch  chiefs) 

146.  Assigned  purchasing/contracting  specialist 

147.  Assigned  administrative  support 

148.  Total  office  experience  level  (years/months) 

149.  Average  office  experience  level  (years/months) 

150.  Total  experience  level  without  administrative  support 

151.  Average  experience  level  without  administrative 
support 

152.  Priority  1-3  CALT 

153.  Priority  4-8  CALT 

154.  Priority  9-15  CALT 

155.  Overall  CALT 


1.  Last  IG  rating. 

2.  Number  of  contracts  behind  schedule  (delinquent). 

3.  Percentage  of  competitive  actions. 

4.  N\imber  of  personnel  assigned  versus  authorized. 

5.  Total  ninober  of  contracting  actions. 

6.  Total  number  of  centralized  actions. 

7.  Total  number  of  decentralized  actions. 

8.  Total  dollars  awarded  competitively. 

9.  Number  of  BPAs  administered. 

10.  The  number  of  administrators  and  buyers  (excluding 
management  and  procurement  clerks). 

11.  The  number  of  clerical  support  personnel. 

12.  The  number  of  management  personnel  (includes  front 
office  branch  or  flight  chiefs,  and  Executive 
Officer/NCO) . 

13.  The  number  of  assigned  civilian  personnel. 

14.  The  number  of  assigned  officer  personnel. 

15.  The  number  of  assigned  enlisted  personnel. 

16.  The  total  office  experience  level. 

17.  The  average  office  experience  level  (total  experience 
divided  by  total  number  of  personnel ) . 

18.  The  total  office  experience  level  without  procurement 
clerk  or  administrative  support. 

19.  The  average  office  experience  level  without  procurement 
clerk  or  administrative  support. 

20.  The  total  niunber  of  line  items  received  priority  1-8. 

21.  The  total  nximber  of  line  items  received  (all 
priorities ) . 
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22.  The  number  of  different  customer  organizations  served 
by  our  office. 

23.  The  total  dollars  awarded. 

24.  The  total  number  of  modifications  executed. 

25.  The  priority  1-3  CALT. 

26.  The  priority  4-8  CALT. 

27.  The  priority  9-15  CALT. 

28.  The  average  (overall)  CALT. 

29.  The  total  number  of  Non- Appropriated  Fund  actions. 

30.  The  total  Non-Appropriated  Fvmd  dollars  awarded. 

31.  The  total  niimber  of  Section  8A  actions. 

32.  The  total  Section  8A  dollars  awarded. 

33.  The  total  number  of  line  items  awarded. 

34.  The  total  number  of  centralized  line  items  awarded. 

35.  The  total  number  of  decentralized  line  items  awarded. 

36.  The  total  centralized  dollars  awarded. 

37.  The  total  decentralized  dollars  awarded. 

38.  The  ratio  of  centralized  actions  divided  by  centralized 
line  items  (measure  of  combining  multiple  line  items 
into  one  action } . 

39.  The  ratio  of  decentralized  dollars  awarded  by  total 
dollars  awarded  (measure  of  decentralizing  workload). 

40.  The  ratio  of  decentralized  line  items  awarded  divided 
by  total  line  items  awarded  (measure  of  decentralizing 
workload) . 

41.  The  ratio  of  total  modifications  executed  divided  by 
total  centralized  actions  awarded. 

42.  The  total  set-aside  actions  awarded. 
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43.  The  total  niunber  of  large  business  actions  awarded  that 
were  available  for  small  business  (a  measure  of 
dissolving  set-asides ) . 

44.  The  total  large  business  dollars  awarded  that  were 
availed>le  for  small  business. 

45.  The  ratio  of  small  business  dollars  awarded  divided  by 
total  dollars  availedsle  for  small  business. 

46.  The  percent  competitive  dollars  measure  (competitive 
dollars  divided  by  total  dollars  awarded. 

47.  The  total  number  of  active  contracts  administered. 

48.  The  total  ntunber  of  active  A&E  and  construction 
contracts  administered. 

49.  The  total  number  of  active  service  contracts 
administered . 

50.  The  total  number  of  active  commodities  contracts 
administered . 

51.  The  ratio  of  priced  actions  awarded  divided  by  unpriced 
actions . 
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Appgn<;^ig  C; _ Management  Survey 


NOTE;  The  survey  questions  used  for  this  research  are 
highlighted  in  bold  print. 


FROM;  AFIT/LSA  29  March  1993 

SUBJECT;  Survey  -  Organizational  Performance  Factors  in 
Operational  Contracting  Squadrons 

TO;  Survey  Recipient 

As  part  of  two  thesis  efforts  at  the  Air  Force  Institute  of 
Technology  (AFIT),  we  are  attempting  to  develop  improved 
management  tools  for  operational  contracting  squadrons.  To 
do  thiS/  the  attached  survey  attempts  to  gather  your 
opinions  and  judgments  about  factors  significant  to  a 
contracting  squadron's  performance.  Copies  of  the  survey 
are  being  sent  to  squadron  commanders,  deputy  base 
contracting  officers,  and  contracting  squadron  first 
sergeants  at  all  bases  within  the  CONUS. 

As  a  senior  manager,  you  are  in  a  unique  position  to  provide 
a  critical  body  of  information  necessary  for  these  thesis 
efforts. 

We  estimate  that  completion  of  the  survey  will  only  take 
about  twenty-five  minutes.  The  opinions  euid  judgment  of 
experienced  personnel  such  as  you,  will  provide  a 
significant  contribution  to  the  success  of  these  studies. 

The  enclosed  survey  was  pilot  tested  by  a  sample  of 
contracting  squadron  personnel.  Based  upon  the  pilot  test, 
the  survey  was 

revised  in  order  to  obtain  all  necessary  data  while 
requiring  a  minimum  of  your  time. 

Please  mark  your  responses  directly  on  the  survey.  No 
coding  sheet  is  required,  ffhen  complete,  please  return  the 
completed  survey  in  the  enclosed  envelope. 

Due  to  deadlines  established  by  AFIT,  euid  the  fact  that 
other  phases  of  both  research  projects  cannot  be  carried  out 
until  analysis  of  the  survey  data  is  complete,  we  would 
appreciate  if  you  complete  and  return  the  subject  survey  by 

3  May  1993. 
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Your  cooperation  in  this  matter  is  greatly  appreciated. 

Your  input  will  provide  great  insight  into  how  operational 
contracting  squadrons  can  improve  their  overall  per£orm^ulce 


MARK  W.  FAHRERKAMP,  CAPT,  USAF 
AFIT  Graduate  Student 


MARK  P.  GARST,  CAPT,  USAF 
AFIT  Graduate  Student 


DOUGLAS  E.  JAMES,  CAPT,  USAF 
AFIT  Graduate  Student 


DERMIS  W.  GROSECLOSE,  ILT,  USAF 
AFIT  Graduate  Student 
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SURVEY 


PARTL  BACKGROUND 

This  section  of  the  survey  obtains  infomoation  about  your  background.  The 
information  requested  is  to  ensure  the  groups  you  belong  to  are  accurately  represented, 
not  to  identify  you  as  an  individual.  Your  anonymity  will  be  maintained  throughout  this 
study. 

1.  Total  months  in  present  job  position. 


2.  Which  of  the  following  applies  to  you? 

1.  Civilian 

2.  OflBcer 

3.  Enlisted 

3.  How  many  years  of  contracting  experience  do  you  have? 


4.  Wha*  i^ior  command  are  you  presently  assigned  to? 

1.  Air  Mobility  Command 

2.  Air  Combat  Command 

3.  Air  Training  Command 

4.  Air  Force  Material  Command 

5.  Other 

5.  How  important  is  it  to  you  that  your  organization  achieve 
optimal  effectiveness? 

1.  Of  no  importance. 

2.  Of  slight  importance. 

3.  [oderately  important. 

4.  Fairly  important. 

5.  Extremely  important. 
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6.  What  was  tlw  last  overall  comnuuid  Inspector  General  (IG)  rating  of 
your  organization? 

1.  Outstanding 

2.  Excdlent 

3.  Satisfiurtory 
.4.  Marginal 

5.  Unsatis&ctory 

6.  Does  not  apply. 

7.  Were  you  in  your  present  job  position  during  the  last  command  IG 
inspection? 

1.  Yes 

2.  No 

Please  indicate  your  opinions  on  the  following  two  statemoits; 

8.  The  current  command  IG  evaluation  system  satisfactorily 
measures  organizational  performance. 

1.  Strongly  disagree. 

2.  Disagree. 

3.  Neutral. 

4.  Agree. 

5.  Strongly  agree. 

9.  An  evaluation  system,  different  from  the  command  IG,  would  be 
useful  in  measuring  overall  organizational  performance. 

1.  Stroi^y  disagree. 

2.  Disagree. 

3.  Neutral. 

4.  Agree. 

5.  Strongly  agree. 


PART  n.  ORGANIZATIONAL  PERFORMANCE  FACTORS 

This  portion  of  the  survey  contains  &ctors  sometimes  used  to  measure 
performance  of  operational  contracting  squadrons.  Please  indicate  the  importance  you 
would  assign  to  each  factor  by  circling  the  appropriate  number  on  the  scale  printed  to  the 
right  of  each  &ctor.  Scale  values  are  shown  below; 

Scale  of  Importance 

Definitely  Not  Somewhat  Not  Somev^^  Definitely 

Important  Important  Neutral  Important  Inq)ortant 


PERFORMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


10.  Number  of  open  purchase  requests.  1  2  3  4  5 

11.  Number  of  Small  Business  goals  achieved.  1  2  3  4  5 

12.  Last  IG  rating.  1  2  3  4  5 

13.  Command  or  higher  contracting  awards  (i.e.  Best  Contracting  1  2  3  4  5 

Squadron  in  ACC,  Professionalism  in  Contracting,  etc.) 

14.  Outstanding  Unit  Citation.  1  2  3  4  5 

15.  Number  of  contracts  behind  schedule  (delinquent).  1  2  3  4  5 

16.  Total  number  of  delinquent  contractors.  1  2  3  4  5 

17.  Number  of  SF  129  packages  mailed.  1  2  3  4  5 

18.  Number  of  new  vendors  identified  and  loaded  in  BCAS.  1  2  3  4  5 

19.  Number  of  Reports  of  Discrepancy  (RODS).  1  2  3  4  5 

20.  Number  of  STEP  promotions.  1  2  3  4  5 

21.  Number  of  protests  received.  1  2  3  4  5 
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Scale  of  Importance 


Definitdy  Not 

Somewhat  Not 

Somewhat 

Definitely 

Important 

1 

Important 

1 

Neutral 

1 

Important 

1 

Important 

1  . 

1 

2 

3 

4 

5 

PERFORMANCE  FACTOR 

22.  Number  of  protests  successfully  defended. 

23.  Total  number  of  ratifications. 

24.  Percentage  of  competitive  actions. 

25.  Number  of  personnel  assigned  versus  authorized. 

26.  Total  number  of  contracting  actions. 

27.  Total  number  of  centralized  actions. 

28.  Total  number  of  decentralized  actions. 

29.  Total  dollars  awarded  competitivety. 

30.  Number  of  interest  payments  paid  due  to  late  processing 
of  contractor  invoice. 

3 1 .  Number  of  contracts  awarded  using  source  selection. 

32.  Number  of  undefinitized  actions. 

33.  Number  of  BPAs  adminbtered. 

34.  Number  of  contracting  ofiBcer  warrants. 

35.  Number  of  solicitation  amendments  due  to  contracting 
personnel  error. 

36.  Number  of  customer  education  classes/training  provided. 


SCALE  OF  IMPORTANCE 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 

1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 

1  2  3  4  5 
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Scale  of  Importance 


D^mitely  Not 

Somewhat  Not 

Somewhat 

Definitely 

Important 

1 

Important 

1 

Noitral 

.1 

Important 

1 

Important 

1 

1 

2 

3 

4 

5 

PERFORMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


37.  Number  of  man-hours  e^qjended  to  support  deployments, 
contingency  contracting  activities,  and  mobility  exercises. 

38.  The  ratio  of  formal  training  quotas  received  divided 
by  the  number  utilized. 

39.  The  number  ofin-house  proficiency  tnuning  performed. 

40.  Number  of  squadron  Operating  Instructions. 

4 1 .  Number  of  self  inspections  conducted. 

42.  Number  of  Value  Engineering  Change  Proposals  received. 

43.  Number  of  Value  Engineering  Change  Proposals  approved. 

44.  Percent  of  the  time  BCAS  is  available  to  users. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


45.  Number  of  Purchase  Requests  over  90  days  old.  1  2  3  4  5 

46.  Number  ofPurchase  Requests  over  120  days  old.  1  2  3  4  5 

47.  Number  of  vendor  follow-ups  due  to  late  delivery.  1  2  3  4  5 

48.  Number  ofpersonnel  qualified  for  at  least  APDP  level  1  certification.  1  2  3  4  5 

49.  Number  ofwalk-through  purchase  requests.  1  2  3  4  5 

50.  The  number  of  administrators  and  buyers  (excluding  1  2  3  4  5 

management  and  procurement  clerks). 


51.  The  number  of  clerical  support  personnel. 


1  2  3  4  5 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1  2  3  4  5 


PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

52.  The  number  of  management  personnel  (includes  front  office  1  2  3  4  5 
branch  or  flight  chiefs,  and  Executive  Officer/NCO). 


53.  The  number  of  assigned  civilian  personneL  1  2  3  4  5 

54.  The  number  of  assigned  officer  personneL  1  2  3  4  5 

55.  The  number  of  assigned  enlisted  personneL  1  2  3  4  5 

56.  The  total  office  experience  leveL  1  2  3  4  5 

57.  The  average  office  experience  level  (total  1  2  3  4  5 


experience  divided  by  total  number  of  personnel). 

58.  The  total  office  experience  level  without  procurement  1  2  3  4  5 

clerk  or  administrative  support 

59.  The  average  office  experience  level  without  1  2  3  4  5 

procurement  clerk  or  administrative  support 

60.  The  total  office  operating  budget  (money  you  have  to  1  2  3  4  5 

operate  the  office). 

61.  The  total  number  ofline  items  received  priority  1-8.  1  2  3  4  5 

62.  The  total  number  of  line  items  received  (all  priorities).  1  2  3  4  5 

63.  The  number  of  different  customer  organizations  served  by  1  2  3  4  5 

our  office. 

64.  The  total  dollars  awarded.  1  2  3  4  5 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 

I _ 1 _ 1 _ ^ _ I _ 

1  2  3  4  5 


PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

65.  The  total  number  of  modifications  executed.  1  2  3  4  5 

66.  The  priority  1-3  CALT.  1  2  3  4  5 

67.  The  priority  4-8  CALT.  1  2  3  4  5 

68.  The  priority  9-15  CALT.  1  2  3  4  5 

69.  The  overall  (average)  CALT.  1  2  3  4  5 

70.  The  total  numberofNon-Appropriated  Fund  actions.  1  2  3  4  5 

71.  The  total  Non-Appropriated  Fund  doUars  awarded.  1  2  3  4  5 

72.  The  total  number  of  Section  8A  actions.  1  2  3  4  5 

73.  The  total  Section  8A  dollars  awarded.  1  2  3  4  5 

74.  The  total  number  ofline  items  awarded.  1  2  3  4  5 

75.  The  total  number  of  centralized  line  items  awarded.  1  2  3  4  5 

76.  The  total  number  of  decentralized  line  items  awarded.  1  2  3  4  5 

77.  The  total  centralized  dollars  awarded.  1  2  3  4  5 


78.  The  total  decentralized  doUars  awarded. 


1  2  3  4  5 


79.  The  ratio  of  centralized  actions  divided  by  centralized  1  2  3  4  5 

line  items  (measure  of  combining  multiple  line  items 
into  one  action). 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1  2  3  4  5 


PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

80.  The  ratio  of  decentralized  dollars  awarded  divided  by  1  2  3  4  5 

total  dollars  awarded  (measure  of  decentralizing  workload). 

81.  The  ratio  of  decentralized  line  items  awarded  divided  by  1  2  3  4  5 

total  line  items  awarded  (measure  of  decentralizing  workload). 

Note  for  question  82:  The  following  ratio  gives  a  percentage  of 

in-house  awards  which  required  modification  (a  low  percentage 

is  good). 

82.  The  ratio  of  total  modifications  executed  divided  by  1  2  3  4  5 

total  centralized  actions  awarded. 

83.  The  total  set-aside  actions  awarded.  1  2  3  4  5 

84.  The  total  number  of  large  business  actions  awarded  that  1  2  3  4  5 

were  available  for  small  business  (a  measure  of 

dissolving  set-asides). 

85.  The  total  large  business  dollars  awarded  that  were  1  2  3  4  5 

available  for  small  business. 

86.  The  ratio  of  small  business  dollars  awarded  divided  1  2  3  4  5 

by  total  dollars  available  for  small  business. 

87.  The  percent  competitive  dollars  measure  (competitive  1  2  3  4  5 

dollars  divided  by  total  dollars  awarded). 

88.  The  total  number  of  active  contracts  administered.  1  2  3  4  5 

89.  The  total  number  of  active  A&E  and  construction  contracts  1  2  3  4  5 

administered. 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1  2  3  4  5 

PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

90.  The  total  number  of  active  service  contracts  administered.  1  2  3  4  S 

91.  The  total  number  of  active  commodities  contracts  1  2  3  4  5 

administered. 

Note  for  question  92:  Priced  actions  represent  actions  that 
were  purchased  fi'om  such  pre-priced  instruments  as  calls  against 
blanket  delivery  orders  or  blanket  purchase  agreements  and 
delivery  orders  against  pre-priced  contracts.  Unpriced  actions 
are  those  actions  which  required  pricing  action  prior  to  award. 

This  includes  items  such  as  purchase  orders,  centralized  BP  As, 
imprest  fiind  actions  and  contracts. 

92.  The  ratio  of  priced  actions  awarded  divided  by  1  2  3  4  5 

unpriced  actions. 

PART  HI:  ADDITIONAL  PERFORMANCE  FACTORS 

Please  add  any  additional  factors  you  have  used  to  evaluate  contracting 
performance  and  indicate  the  importance  of  each. 


93. 

94. 

95. 

96. 

97. 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
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PART  IV.  OVERALL  PERFORMANCE 


Using  the  factors  listed  above,  including  the  factors  you  may  have  added  in 
questions  93-97,  select  the  five  you  feel  are  the  most  important  in  defining  organizational 
performance  within  an  operational  contracting  squadron.  Indicate  your  ranking  of  these 
five  factors  by  inserting  their  question  number  in  the  blanks  below. 

FIRST  in  SECOND  in  THIRD  in  FOURTH  in  FIFTH  in 
Importance  Importance  Importance  Importance  Importance 


PARTV.  OVERALL  PERFORMANCE  OF  YOUR 
SQUADRON/ORGANIZATION 

Based  upon  the  following  scale,  how  would  you  rank  the  overall  performance  of 
the  contracting  squadron/organization  you  are  assigned.  This  question  is  based  solely  on 
your  belief,  not  the  last  IG  rating  or  command  evaluation.  Please  circle  the  appropriate 
level  of  performance. 


Unsatisfactory  Marginal  Satisfactory  Excellent  Outstanding 
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PART  VX  DESIRABLE  CHARACTERISTICS  OF  A  PERFORMANCE 
EVALUATION  AND  FEEDBACK  SYSTEM 

This  part  of  the  survey  is  designed  to  obtain  managers  views  and  desires 
concerning  a  performance  evaluation  and  feedback  system  for  operational  contracting.  In 
addition,  this  section  contains  questions  concerning  current  methods  of  evaluating  and 
providing  feedback  to  the  operational  contracting  squadron.  Please  indicate  the 
importance  you  would  assign  to  the  statements  listed  below.  Continue  using  the  following 
scale. 


Scale  of  Importance 

Definitely  Not  Somewhat  Not  Somewhat  Defimtely 

Important  Important  Neutral  Important  Important 

_ I _ I _ I _ I _ I _ 


1  2  3  4  5 

PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

98.  The  current  IG  method  of  performance  evaluation  provides  1  2  3  4  5 
feedback  to  help  improve  performance  which  is: 

99.  Current  command  and  Air  Force  awards  provide  feedback  1  2  3  4  5 
concerning  performance  which  is: 

100.  Improving  operational  contracting  squadron  productivity  1  2  3  4  5 

and  efficiency  is  a  goal  which  is: 

101.  A  contracting  evaluation  system  which  simultaneously  1  2  3  4  5 

evaluates  several  inputs  and  outputs  to  the  process 

(as  opposed  to  single  measures  such  as  CALT),  is  a  tool 
which  is: 

102.  A  contracting  evaluation  system  which  compares  all  1  2  3  4  5 

operational  contracting  squadrons  while  taking  into 

account  differences  in  squadron  characteristics  (such  as 
manning,  experience,  and  workload),  is  a  tool  which  is: 

103.  A  contracting  evaluation  system  which  does  not  rely  1  2  3  4  5 

solely  upon  measures  which  the  chain  of  command 

(LG,  Wing  CC)  find  important,  is  a  tool  which  is: 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1  2  3  4  5 


PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

104.  An  evaluation  system  which  compares  squadrons  relative  1  2  3  4  5 

to  other  contracting  squadrons,  is  a  tool  which  is: 

105.  A  contracting  evaluation  system  which  compares  contracting  1  2  3  4  5 
squadrons  to  the  best  performers  as  opposed  to  the 

average,  is  a  tool  which  is: 

106.  A  contracting  evaluation  system  which  provides  managers  1  2  3  4  5 
with  timely  feedback,  including  exact  data  on  resource 

utilization  and  relative  efficiencies  compared  with 
other  squadrons,  is  a  tool  which  is: 

PART  m  ADDITIONAL  COMMENTS  ON  CONTRACTING  PERFORMANCE 

Using  the  space  provided  below,  please  feel  fi-ee  to  add  any  additional  comments 
you  may  have  concerning  the  performance  of  operational  contracting  squadrons. 
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DERIVATION  OF  HCU  VALUES  FOR  DYE8S  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  GEORGE  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  CANON  AFB 


DERIVATION  OF  HCU  VALUES  FOR  HOLLOMAN  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  LANGLEY  (BASE)  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  McCONNELL  AFB 
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DEWVATION  OFHCU  VALUES  FOR  OFFUT  (390»th)  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  HOMESTEAD  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  Kl  SAWYER  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  GRAND  FORKS  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  LORING  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  POPE  AFB 
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DERIVATION  OF  HCU  VALUES  FOR  NELLIS  AFB 
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